Atmospherian's Dev Blog

So lately i’ve been working on creating a more formal website for myself to highlight my design work and and show potential clients what i can do for them. This site is being built in Flash using FlashDevelop and the Flex SDK. I am not using the Flash IDE from Adobe at all. The entire site is being written in Actionscript 3, including the animations. The site will consist of three main sections: the marketing site which will contain my portfolio and services, the client portal which will be an area where clients can view the status of their project, keep track of how much time is being spent, communicate with the developers (currently only me), upload and view documents and a few other features yet to be determined. The third part will be the admin section where i can manage all of the clients logins, post updates, clock im hours that i have worked and so on.

The site is being powered on the back end by Ruby on Rails using a very basic REST api. So far the client portal is about 50% complete, which has been taking some time because i can including a few different types of charts and graphs that i am writing from scratch in AS3 which is needless to say time consuming.

The more i used AS3 the more i began to realize that although certain things about the language are nice (static/dynamic typing, ability to import SWC libraries), others felt a bit lacking (no generics, enums, anonymous types). I noticed that in FlashDevelop there was this other language called haXe that it supported. Intrigued, i looked it up and quickly read over the language reference and knew that this would be a great language to develop Flash projects in. I was especially glad to see that the bytecode generated by the haXe compiler for the flash vm is often times faster than the code generated by the Adobe compilers. This is especially important for flash game developers who would like to use the nice 3D engines that are available.

One such 3D engine, FFilmation (ffilmation.org) is open-source and looks rather nice at this point. I have been working on the design of an online game based on flash that this engine would work nicely in. The problem is, the engine is written in AS3, and of course i am going to be using haXe for this project. My first thought was to manually translate the AS3 code into haXe, but once i took a closer look at the source code and it’s many files and all the work i would have to do to make the translation i thought there must be a better way. I looked online for an as3tohaxe converter and found only one that seems like it could work. When i ran this converter on the ffilmation directory, it did indeed spit out .hx files, but there was only the most basic of transformations made (basically converting void to Void, int to Int, Number to Float) the basic differences between AS3 and haXe. But the true list of differences is a bit longer:

AS3 supports wildcard import statements, haXe does not. Classes in AS3 can start with lowercase letters, in haXe thet must be uppercase.

Those two restrictions would make it very annoying to have to translate everything manually. So i figured if i wanted this library to be written in haXe, instead of spending all this time doing the translation, that would have to be updated manually every time the original AS3 source was updated, i could spend the time writing a better AS3 to haXe translator that more closely inspects the input files, determines what classes are being used, re-generates the import statements, refactors the class names, converts the data types, converts the ‘for’ loop syntax, and converts the package block into a statement.

After a few days of design and coding i have a working prototype that does everything listed above. I wrote it in haXe and its running on the neko vm so it should be cross-platform and compile on windows, linux and mac osx. I am hosting the code on GitHub so anyone can feel free to fork it and try it out, report any bugs, and ideally patches to make it better. I know its a bit rough right now and i’m sure the code is quite messy, but it will get better.

Right now i am analyzing the output that is being generated to see how things look and see where things need to be improved. My goal is to have the haXe code compile without any modifications needing to be made directly out of the translator.

Look forward to my next few posts which will cover what my analysis results are and what the next steps will be for the project.

So i’ve beena bit quiet about InterIMAP lately, but that doesn’t mean i haven’t been working on it. Today i will be checking in the latest update of the code which includes some really cool functionality, as well as a new IMAPShell project.

Code Updates

• Added support for retrieving and parsing message structure and content
• Fix some locking issues in the DataManager to make multi-threading work better
• Added support for retrieving and updating message flags
• Added a Log Aggregator that collects the logging information from all worker threads
• Improved the parsing of email addresses
• Added support for decoding message content in the correct format
• Added support to all requests to monitor the amount of data received to facilitate progress indicators
• Added the FullMessageRequest which will asynchronously retrieve the header, structure and all parts of a message while providing progress indication.
• Added lots of functionality to the GUITest app including a message viewer to test how a message will appear.

IMAPShell Project

There is a new project in the solution called IMAPShell, and as the name implies, it is a console command shell that allows CLI style interaction with the IMAP library. The goal of the project is to provide a quick and easy way to access message information without having to configure an email client.

It is in its early stages but you can already connect to a server, navigate the directory structure of a mailbox using dir/cd commands and list the messages in a folder using the list command. Command help is fully integrated.

What’s Next

Here is a list of what i will be working on for the next update:

• Support for creating/deleting folders
• Support for copying/moving messages
• Support for saving/loading messages and attachments to a local file
• Support for easily detecting new messages in any folder

This post will serve as the official requirements documentation for the InterIMAP library. Please comment on this post with your suggestions for further requirements.

Core Requirements

Storage Requirements

API Requirements

• Provide clean and clear interface to IMAP protocol
• Provide easy to use Request/Response system
• Provide clean object model

So i’ve been reading Steve McConnell’s Code Complete lately, and it’s got me thinking quite a bit about the new work i’ve been doing on the InterIMAP library. When i started working on the new implementation i had this grand view of a well thought-out, well designed system, and to a certain extent, i feel i accomplished that, but in some areas, i fell short. 

I initially felt that the new system would only ever need to connect to one server at a time, and as a result i wrote all the manager classes as singletons. I have since come to the conclusion that it may be necessary for multiple servers/accounts to be accessed concurrently, so that will be one of the major requirements going forward. Additionally, it may also be important to some people to be able to store the information retrieved from a mailbox in a full fledged RDBMS, as well as a “local cache” type system, so i will also be working that into the new architecture.

My next post will be a summary of what the requirements of the library will be. Please leave comments as to what you would like to have available in the library.

The time has come for me to continue work on InterIMAP. I have many changes and improvements planned for this library that will improve its usability. In an effort to flesh out my ideas and implementation, i will be putting together different posts for each topic that needs exploration. I welcome all comments and suggestions from those few of you who find their way here. 

To kick things off, here is a partial list of the features i am planning to implement or improve (in order of priority):

• Better code structure utilizing more standard patterns
• Completely Asynchronous client API
• Improved offline cache system based on a ‘mailbox’ file that will contain the configuration information, messages and attachments in a zip format. This will allow an entire mailbox to be portable between different systems.
• Build a basic graphical client to demonstrate the new asynchronous API and other features
• Compatibility with Mono so the library can be used on Linux, Mac OS X, and any other supported platforms. 
• Convert the graphical configuration editor into a component that can be dropped into any application.
• Build a second configuration component based on WPF that can be easily customized. 

My next post will go into detail about what i plan to do about the code structure and how i intend to refactor, and reorganize everything.

I am very excited to let you guys know that the re-engineering of the message processing architecture is currently stable and working. My new system for processing message content has not only reduced the number of send/receives from the server to a maximum of 3, but has reduced the code of IMAP.cs by roughly 50-60% making it much cleaner and easier to read and understand.

The original way of processing the message content, which was written by the original author of the code, Rohit Joshi, worked well, except for the fact that it choked on many messages from Gmail, which as i understand is one of the main services that this library is being used with (Exchange is the other). Modifying his code to work with gmail proved nearly impossible due to the extreme variations that can occur in body structure. The process involved first getting the BODYSTRUCTURE of the message, parsing that and then getting each individual BODY[x(.x.x)] section. While this is a valid solution, it is very difficult to write something that can parse the BODYSTRUCTURE in its seemingly infinite variations.

The solution is to pull the full content of the message directly using the BODY[] parameter. The result of this command is every section of the message being return in sequence, usually demarcated with a ‘boundary’ value (except in the case of messages with a single text/plain content type). After this boundary, is the header meta data for that section, which can include the content-type, transfer encoding, disposition, etc. making it easy to parse this header data and properly construct IMAPMessageContent objects.

What does this mean for you? it means that there is now a complete, accurate, and efficient process in place for retrieving the message content from both gmail and exchange.

So whats next? Well, during testing, it became clear to me that for large accounts, having all of the message attachments stored in memory could require a significant amount system resources, much more than an email system should need. So with that in mind i am starting to think about ways to store attachments outside of the main cache file, so that when the cache is loaded, the attachments arent loaded until they are requested.

Major updates in the works this week. After some intial testing with GMail, several show-stopping bugs were discovered in the body structure processing code. Modfying the existing code wasn’t possible as the differences were too severe. Instead i opted to re-write the entire message parsing system to be more flexible and to directly interact with the object model, as opposed to generating XML data first and then using a DataSet to read in that XML which was a cheap hack i employed to get to the message data.

I am not quite ready to upload the newest code as i want to perform some more testing with gmail and exchange before i feel it is ready for a wider audience. There is one bug in particular that is show-stopping with gmail that i am trying to address. Once i get that cleared up we should be in good shape.

Stay tuned for more updates as testing on the re-write continues…

New updates coming this week:

Simple and Advanced Searching
I have added a new search system to the library that should make searching within a folder very simple. There are two new objects in the namespace, IMAPSearchQuery and IMAPSearchResult. The SearchQuery object contains many properties that represent the various search options that can be specified.

The actual Search method is defined in IMAPFolder and takes a IMAPSearchQuery instance as a parameter. The search method returns a IMAPSearchResult object that contains a reference to the query that was used, the folder that was searched and a list of IMAPMessage objects that are the results.

To simplify the search process, several static methods have been defined in the IMAPSearchQuery class that take a single argument, and return an IMAPSearchQuery object, which can be plugged directly into the search method like this:

IMAPFolder f = client.Folders["INBOX"];
IMAPSearchResult sResult = f.Search(IMAPSearchQuery.QuickSearchSubject(“IMAP”));

What the above code basically does is first gets the folder named INBOX, and then calls the Search method on that folder using the one of the quick search static functions to automatically create an IMAPSearchQuery instance to use for the search. The results of the search are stored in sResult and the messages that match the search query can be found in sResult.Messages.

For more advanced searching you can create a IMAPSearchQuery object manually, and specify the terms that you wish to search for. You can specify values for any of the properties that are defined in that class and each will be taken into account during the search.

Currently this system will work only for including all the messages that match the query. There is no support for ‘OR’ or ‘NOT’ style parameters as of yet. The plan i have for this is to add special List<IMAPSearchQuery> properties to the query object, one for ‘NOT’ and one for ‘OR’ where new SearchQueries can be added and processed and included into the main query. I have to experiment with this option and see if it will work well and be intuitive and easy to use.

Local Data Cache
I’ve spent a great deal of time this week working on the local cache system. I’m happy to say that it’s nearly complete. Here is how it currently works:

First, to enable the local cache the developer needs to specify a cache file and cache format in the configuration. This can either be done at runtime, or using the configuration generators.

In the constructor for the IMAPClient class, the configuration is checked to see if a file is defined. if it is, a flag is set, and if autologon is also specified, the system logs in.

Once logged in, the system checks if the specified cache file already exists. If it does, it loads the cache into the object model, and then (if not in offline mode) the system syncs the cache with server (a config option has been added to set the system to auto sync or not). Updates are one-way with the server being the authority. (If the system is running in offline mode, any functions that would change the folder contents or structure are disabled). If the file does not exist, the system then begins the process of building a new cache file. Please be warned that if you are connecting to a large account with many folders and messages this process can definitely take some time (several hours in the worst cases, depending on the location of the server relative to your location, ie. if the server is on a local LAN it will be faster than pulling from the internet).

Once the cache is built, all the data will already be in the object model so no prep work is required to start accessing the messages.

When working in online mode, all of the message and folder copy/move/delete methods will also update the cache when their process completes successfully. This update should not force any special server commands to be run, it should simply serialize the current state of the object model.

Note: this post will be updated as the current state of the feature set changes.

Current Features

• Dynamically loaded message data
• Caching of message data including attachments to filesystem
• Supports SSL connections
• Ability to run in offline mode when using caching system
• Ability to create and delete folders and copy/move/delete messages
• Support external configuration files for all options
• Includes Windows and command-line tools for creating and managing configuration files
• Provides advanced search system that works in both online and offline modes
• Ability to save attachments to local filesystem

Planned Features

• Nested search queries using NOT and OR operators
• Include simple e-mail client to demonstrate use of the library

Here is a break down of this weeks updates to the library:

Configuration Editor (Window app)
I created a simple Windows application that can be used to create and edit configuration files for the InterIMAP library. This should make it much easier to create and manage configuration files.

Local Data Cache Update
So the local data cache has been coming along pretty well. I can now successfully save and load data from the cache and use it to populate the object model upon startup. This is great, however there is much left to be done. For example, there is no synchronization, so if something has changed on the server (folders added/deleted, messages added/deleted), the entire cache would have to be re-generate and saved again. The next step is to design a way to check for changes on the server and changes to the local cache and decide which changes take precedence. Should the local changes be applied first, and then pull the new state of affairs from the server? or should the server be more authoritarian and overwrite the changes made locally when in offline mode?

I implemented a few message management functions on the folder level that allow you to copy, move and delete a message. One problem i ran into though, if the cache is not updated after performing these operations, when i re-ran my test app, messages were loaded that no longer exist on the server and should also not exist in the cache.

Right now the SaveCache method forces every message to be downloaded from the server before everything is saved. But it should behave differently if there is already data from the cache loaded. It shouldnt need to re-download messages that are already stored. Part of the problem i see is that due to the dynamic data loading, most messages by default dont have the content loaded unless the message content has already been accessed elsewhere in the app before the cache is saved.

I will continue to research this and blog more about it as development continues

Bugs Fixed
I Received an email from a developer who was trying to use the SSL functionality to connect to Gmail and got an exception when the system tried to examine the ‘[Gmail]‘ folder. The problem was this folder has the \Noselect flag specified which the system wasn’t told to look for and skip that folder if it was found. This has been fixed so the folder processing method will now skip folders that are flagged as not selectable (this applies to EXAMINE as well).

Once that bug was solved, another became readily apparent. After some frustrating debugging and searching i discovered what was causing the ReceiveBuffer method to hang when trying to receive the data from the message body using the SSL connection. I was trying to read from the SslStream object directly instead of the StreamReader object, which both have a Read() method, although slightly different. This has been corrected so you should not have any problems reading data from an SSL connection now.