Atmospherian's Dev Blog

So within the last few minutes i figured out why i was seeing wrong results when processing the flags for a message. Originally i was getting the flags as part of the message header using an IMAP command like this:

UID FETCH [message uid here] (FLAGS BODY[HEADER])

And it did return the flags, but not accurate ones. In the case of GMail, for every message, regardless of its read/unread status in the UI, the \Seen flag would be returned. This coupled with the fact that the default value for the Seen flag in the Message was true, resulted in every message appearing as though it is new. Not cool.

The solution was to request the message flags separate from the header like this:

UID FETCH [message uid here] FLAGS

Sending this command returns the correct flags, or in the case of a new message, no flags. The only drawback to his approach is the fact that another roundtrip to the server must be performed for each message, minor additional overhead, but additional overhead nonetheless.

Now that the new message detection code is working i can implement a nice quick method of checking for new messages with appropriate callbacks and whatnot. Follow me on twitter @Atmospherian to get updates when i check-in the code.

So i haven’t been working on Haxer too much lately, mostly because i’ve hit a wall as far how to best convert some AS3 code to Haxe without having to implement a full parser/lexer. The main area i am struggling with is converting “o is Object” and “o as Object”  which sounds trivial at first, but when you look at the myriad of possible ways those statements can appear in code, it becomes a much more complex task.

In HaXe, the “is” operator/keyword/whatever it is, is written as “Std.is(obj, type)”  and the “as” operator is written as “cast(obj, type)”. This presents quite a challenge when you have lines like this in AS3:

There are other, more complicated examples that make using regular expressions nearly impossible. I am open to a discussion as to how i might be able to implement such a conversion.

I know it’s been a while since i’ve posted about InterIMAP and i want to let everyone know that i am planning to continue work on it, fixing bugs and adding features.

Based on the issues and discussions that have been posted on the codeplex site, it seems that some people are still using the old code base in the Synchronous folder tree. This branch will be marked as deprecated going forward as i will not be maintaining it anymore. The asynchronous branch contains far better code, and even supports performing operations in a synchronous manner (blocking the calling thread until the operation completes) so there is no reason to use the old code.

Through some basic testing i did on the code to see what state it is in, i noticed that the new message flag detection code was not working properly, marking messages as new that weren’t necessarily so. This will be the first area I will work on correcting as i recall the message flags that the server returns can vary from server to server (as most things with IMAP tend to be). Once i can correctly determine if a message is new or not i will work on implementing a quick and easy way of checking for new messages either in all folders, or in a specific folder only.

I encourage everyone who uses the library to submit any requests or bugs they find either on this blog or on the codeplex site so that they can be incorporated into the code.

With the rewrite of my as3 to haxe translator in Java, i have also renamed the program ‘Haxer’. It just sounds a whole lot cooler. The current version of the code is up on git hub. Right now it only knows how to skip comments, detect and convert types correctly. I *think* i’ve managed to nail down the best way to do the type refactoring without having to loop over everything a bunch of times. So far the results look promising.

I’ve been trying to keep an up to date list of all the differences between AS3 and haXe when it comes to translating. Here is the list as it stands now:

• Each type must being with an uppercase letter
  • This causes issues when classes are defined in AS3 with a lowecase character as the first character. The class definition and all references must be refactored.
• Each type being used must be explicitly imported, haXe does not support wildcard imports.
  • This requires that every type being used in a class must be detected so that the import statements can be regenerated.
• Package definitions must be changed from a block to a single statement
• For loops must be converted to the ‘…’ syntax
• All variables must be strongly typed, or use Dynamic
• A class constructor name must be converted to ‘new’
• The syntax of properties is very different. get and set are not access modifiers to functions in haXe
  • Considering that properties in AS3 are implemented as two unrelated functions, being able to correctly convert them would be difficult.
• void => Void
• int => Int
• Number => Float
• Array => Array[Dynamic]
• Boolean => Bool
• Class definitions in haXe do not take any access modifiers

If anyone knows of any other differences i should be aware of, please let me know in the comments. thanks.

So over the last few days i’ve been looking at the output that was being generated by my first attempt at writing an As3tohaxe translator. The results have been inconsistent to say the least. Not inconsistent in terms of different results everytime i ran it, but inconsistent within a file itself. Some variable definitions would get updated, while others would not. Some for loops would be translated, and others wouldn’t. Now normally it would be easy to step though the code and see why this is happening, but in haXe and neko there is no debugger. The only thing you can do is litter your code with println’s all over the place, which isn’t really all that helpful.

I thought it would be cool to write the program in the target language of its output, but given this rather significant shortcoming to the development environment, something more robust and mature is needed. Enter Java and Scala. Combined with IntelliJ (my favorite Java IDE) I’ve begun re-writting the program with the help of the built-in debugger. I thought it would be interesting to take this opportunity to learn Scala at the same time, but that is proving to be a bit challenging as Scala’s syntax flirts with both OO and Functional paradigms, it’s proving a bit difficult to wrap my head around. So for the time being the code is a mix of Scala and Java while i learn more about Scala and how to apply its strengths to this project. As Steve McConnell would say, you have to program into your language, not in it. What he means is using the features of the language you are writting in to make accomplishing your goals easier. Instead of applying the same algorithms and methodologies with just different syntax.

So with this new approach in mind i am now in the process of re-designing the program and will hopefully have some better results to report in a few days.

So lately i’ve been working on creating a more formal website for myself to highlight my design work and and show potential clients what i can do for them. This site is being built in Flash using FlashDevelop and the Flex SDK. I am not using the Flash IDE from Adobe at all. The entire site is being written in Actionscript 3, including the animations. The site will consist of three main sections: the marketing site which will contain my portfolio and services, the client portal which will be an area where clients can view the status of their project, keep track of how much time is being spent, communicate with the developers (currently only me), upload and view documents and a few other features yet to be determined. The third part will be the admin section where i can manage all of the clients logins, post updates, clock im hours that i have worked and so on.

The site is being powered on the back end by Ruby on Rails using a very basic REST api. So far the client portal is about 50% complete, which has been taking some time because i can including a few different types of charts and graphs that i am writing from scratch in AS3 which is needless to say time consuming.

The more i used AS3 the more i began to realize that although certain things about the language are nice (static/dynamic typing, ability to import SWC libraries), others felt a bit lacking (no generics, enums, anonymous types). I noticed that in FlashDevelop there was this other language called haXe that it supported. Intrigued, i looked it up and quickly read over the language reference and knew that this would be a great language to develop Flash projects in. I was especially glad to see that the bytecode generated by the haXe compiler for the flash vm is often times faster than the code generated by the Adobe compilers. This is especially important for flash game developers who would like to use the nice 3D engines that are available.

One such 3D engine, FFilmation (ffilmation.org) is open-source and looks rather nice at this point. I have been working on the design of an online game based on flash that this engine would work nicely in. The problem is, the engine is written in AS3, and of course i am going to be using haXe for this project. My first thought was to manually translate the AS3 code into haXe, but once i took a closer look at the source code and it’s many files and all the work i would have to do to make the translation i thought there must be a better way. I looked online for an as3tohaxe converter and found only one that seems like it could work. When i ran this converter on the ffilmation directory, it did indeed spit out .hx files, but there was only the most basic of transformations made (basically converting void to Void, int to Int, Number to Float) the basic differences between AS3 and haXe. But the true list of differences is a bit longer:

AS3 supports wildcard import statements, haXe does not. Classes in AS3 can start with lowercase letters, in haXe thet must be uppercase.

Those two restrictions would make it very annoying to have to translate everything manually. So i figured if i wanted this library to be written in haXe, instead of spending all this time doing the translation, that would have to be updated manually every time the original AS3 source was updated, i could spend the time writing a better AS3 to haXe translator that more closely inspects the input files, determines what classes are being used, re-generates the import statements, refactors the class names, converts the data types, converts the ‘for’ loop syntax, and converts the package block into a statement.

After a few days of design and coding i have a working prototype that does everything listed above. I wrote it in haXe and its running on the neko vm so it should be cross-platform and compile on windows, linux and mac osx. I am hosting the code on GitHub so anyone can feel free to fork it and try it out, report any bugs, and ideally patches to make it better. I know its a bit rough right now and i’m sure the code is quite messy, but it will get better.

Right now i am analyzing the output that is being generated to see how things look and see where things need to be improved. My goal is to have the haXe code compile without any modifications needing to be made directly out of the translator.

Look forward to my next few posts which will cover what my analysis results are and what the next steps will be for the project.

So i’ve beena bit quiet about InterIMAP lately, but that doesn’t mean i haven’t been working on it. Today i will be checking in the latest update of the code which includes some really cool functionality, as well as a new IMAPShell project.

Code Updates

• Added support for retrieving and parsing message structure and content
• Fix some locking issues in the DataManager to make multi-threading work better
• Added support for retrieving and updating message flags
• Added a Log Aggregator that collects the logging information from all worker threads
• Improved the parsing of email addresses
• Added support for decoding message content in the correct format
• Added support to all requests to monitor the amount of data received to facilitate progress indicators
• Added the FullMessageRequest which will asynchronously retrieve the header, structure and all parts of a message while providing progress indication.
• Added lots of functionality to the GUITest app including a message viewer to test how a message will appear.

IMAPShell Project

There is a new project in the solution called IMAPShell, and as the name implies, it is a console command shell that allows CLI style interaction with the IMAP library. The goal of the project is to provide a quick and easy way to access message information without having to configure an email client.

It is in its early stages but you can already connect to a server, navigate the directory structure of a mailbox using dir/cd commands and list the messages in a folder using the list command. Command help is fully integrated.

What’s Next

Here is a list of what i will be working on for the next update:

• Support for creating/deleting folders
• Support for copying/moving messages
• Support for saving/loading messages and attachments to a local file
• Support for easily detecting new messages in any folder

This post will serve as the official requirements documentation for the InterIMAP library. Please comment on this post with your suggestions for further requirements.

Core Requirements

Storage Requirements

API Requirements

• Provide clean and clear interface to IMAP protocol
• Provide easy to use Request/Response system
• Provide clean object model

So i’ve been reading Steve McConnell’s Code Complete lately, and it’s got me thinking quite a bit about the new work i’ve been doing on the InterIMAP library. When i started working on the new implementation i had this grand view of a well thought-out, well designed system, and to a certain extent, i feel i accomplished that, but in some areas, i fell short. 

I initially felt that the new system would only ever need to connect to one server at a time, and as a result i wrote all the manager classes as singletons. I have since come to the conclusion that it may be necessary for multiple servers/accounts to be accessed concurrently, so that will be one of the major requirements going forward. Additionally, it may also be important to some people to be able to store the information retrieved from a mailbox in a full fledged RDBMS, as well as a “local cache” type system, so i will also be working that into the new architecture.

My next post will be a summary of what the requirements of the library will be. Please leave comments as to what you would like to have available in the library.