OSNews: http://www.osnews.com/story/20212/Book_Review_The_Book_of_IMAP Exploring the Future of Computing en-us Copyright 2001-2013, David Adams adam+nospam@osnews.com Fri, 24 May 2013 10:55:33 GMT http://www.osnews.com/images/osnews.gif OSNews.com http://www.osnews.com My rants with the IMAP protocol and IMAP servers http://www.osnews.com/thread?327917 http://www.osnews.com/thread?327917 I use IMAP and Gmail and I think IMAP has to evolve...It shouldn't limit itself to just a simple remote folder and message manager.

1. It's slow. It wasn't designed for our gigabyte email accounts. When I have to sync my computer with my server, it synces one mail after the other. I have hundreds of thousands of email, it takes a huge amount of time. It should create a single zipped file of the headers and send it to my computer, it would be much faster than unzipped individual files. With numerous three-way TCP/IP connections, syncing with IMAP is just too slow.

2. Remote search is weak. Even with Dovecot. Try performing an advanced search and not only is it slow but often you don't get what you're searching because your IMAP server doesn't index attachments, sometimes it only searches headers and discards bodies.

3. No support for tags. While some MUA and webmails now migrated from folders to tags or labels, IMAP still uses folders, so for instance in Gmail I have "Projects" and "To do" tags, if a message has both labels, in IMAP it is physically duplicated.

4. No support for calendar/address book/RSS. I know IMAP is not a calendar server application but nowadays, many MUAs have calendar, RSS and address book integrated and it's essential to be able to sync a calendar, feeds and an address book without the need to set up an LDAP server.

5. Be in par with new needs. IMAP should offer all one expects from a mail protocol these days. I think when you use Gmail and IMAP, you shouldn't lose functionality, power, features and speed. It's time to dramatically upgrade the IMAP protocol, in the meantime, some people are migrating to web applications that have fast, synced, database-indexed emailing, calendar, RSS and address book.Edited 2008-08-26 06:35 UTC Tue, 26 Aug 2008 06:31:00 GMT donotreply@osnews.com (Liquidator) Comments RE: My rants with the IMAP protocol and IMAP servers http://www.osnews.com/thread?327920 http://www.osnews.com/thread?327920 I am not sure whether you know what you are talking about.

Here's is one for you to try on YOUR machine: zip up a few hundred thousand mini-KB files and report back to us how long it took you. Then delete a random 50000 files and ad 70000 new one into that zip-file; then report back and tell us how long that took. Then do your math and let us know what will happen if the server has to do that to any number of people typically accessing their mail at a time... the server would explode ;) Tue, 26 Aug 2008 08:08:00 GMT donotreply@osnews.com (Googol) Comments RE[2]: My rants with the IMAP protocol and IMAP servers http://www.osnews.com/thread?327921 http://www.osnews.com/thread?327921
I am not sure whether you know what you are talking about.


No comment.

Back to topic. I think the problem you're exposing is that IMAP servers work with *files*. Obviously, email servers would cripple if they had to zip individual small files. And they already cripple just retrieving individual messages. How many times do we get timeout error messages from Squirrelmail when we have too many messages in an IMAP folder? The solution is retrieving relevant messages from a *database*, storing them into a temporary file (single file), zipping this file and sending it to the MUA. This would solve our problem and servers would be happy. Tue, 26 Aug 2008 08:48:00 GMT donotreply@osnews.com (Liquidator) Comments RE[3]: My rants with the IMAP protocol and IMAP servers http://www.osnews.com/thread?327923 http://www.osnews.com/thread?327923 There already are imap servers that use databases,
one of them is archiveopteryx http://www.archiveopteryx.org/

It uses Postgresql for backend storage, this means that you can make use of the extremly quick Postgresql full text search engine to search your mail. Tue, 26 Aug 2008 09:06:00 GMT donotreply@osnews.com (unoengborg) Comments RE[4]: My rants with the IMAP protocol and IMAP servers http://www.osnews.com/thread?327924 http://www.osnews.com/thread?327924 The database is only used for backend storage. DBMail works this way for instance. But the client-server communication doesn't benefit from it. Tue, 26 Aug 2008 09:14:00 GMT donotreply@osnews.com (Liquidator) Comments RE: My rants with the IMAP protocol and IMAP servers http://www.osnews.com/thread?327929 http://www.osnews.com/thread?327929 i haven't done a start from scratch sync for a while, so i dunno if this is already the case.

However instead of the zip option perhaps being able to download multiple emails at the same time (parrallel download)

I do agree that IMAP and also Microsoft's implementation a la Exchange needs improving with the points you have listed. Email is a standard, many companies are using it more than telephone conversations. As commented before many people are reaching multi GB sizes, many people live in their email system, in some cases i wonder if the email system is becoming an operating system in itself. Tue, 26 Aug 2008 10:51:00 GMT donotreply@osnews.com (REM2000) Comments IMAP flags http://www.osnews.com/thread?327930 http://www.osnews.com/thread?327930 Google Mail could use IMAP-tags:
http://www.ii.com/internet/messaging/imap/isps/#keywords Tue, 26 Aug 2008 11:30:00 GMT donotreply@osnews.com (testerus) Comments RE: My rants with the IMAP protocol and IMAP servers http://www.osnews.com/thread?328062 http://www.osnews.com/thread?328062
1. It's slow. It wasn't designed for our gigabyte email accounts. When I have to sync my computer with my server, it syncs one mail after the other.


You're measuring the usage of a single IMAP client with a single IMAP server, and complaining that everything related to IMAP sucks? Yeah, that makes sense.

GMail uses IMAP over SSL, so you have an encryption/decryption process on top of the transport., slowing things down, depending on your CPU.

You don't specify which IMAP client you are using. Accessing GMail from KMail is nice and snappy, as KMail does multi-threaded message downloads/folder syncs.

Perhaps you should investigate other IMAP clients?

I have hundreds of thousands of email, it takes a huge amount of time. It should create a single zipped file of the headers and send it to my computer, it would be much faster than unzipped individual files. With numerous three-way TCP/IP connections, syncing with IMAP is just too slow.


I have 400 MB in my GMail account. Connecting to it from a brand new PC takes less than 5 sec to bring up the inbox (only new messages) and less than a minute to bring up the archive (just under 10,000 messages).

IOW, get a better IMAP client.

2. Remote search is weak. Even with Dovecot. Try performing an advanced search and not only is it slow but often you don't get what you're searching because your IMAP server doesn't index attachments, sometimes it only searches headers and discards bodies.


Get a better IMAP server. Remote search against Cyrus servers is very fast, even against my 1.5 GB work account. I can search the body of all my messages in under 5 minutes, the headers in under 1. Gotta love that server-side indexing.

Zimbra's Cyrus implementation in their OSS version is also quite speedy. And their Network edition is even faster.

3. No support for tags. While some MUA and webmails now migrated from folders to tags or labels, IMAP still uses folders, so for instance in Gmail I have "Projects" and "To do" tags, if a message has both labels, in IMAP it is physically duplicated.


Can't comment on this, as I don't use tagging in any of my IMAP clients.

4. No support for calendar/address book/RSS. I know IMAP is not a calendar server application but nowadays, many MUAs have calendar, RSS and address book integrated and it's essential to be able to sync a calendar, feeds and an address book without the need to set up an LDAP server.


Get a better IMAP server. For instance, Kolab uses Cyrus IMAP to store e-mail, contacts, and calendars in IMAP folders.

5. Be in par with new needs. IMAP should offer all one expects from a mail protocol these days.


It does. Since when is calendars considered part of a "mail" protocol? Or even a "message" protocol"? ;) Edited 2008-08-26 23:20 UTC Tue, 26 Aug 2008 23:18:00 GMT donotreply@osnews.com (phoenix) Comments RE[2]: My rants with the IMAP protocol and IMAP servers http://www.osnews.com/thread?328110 http://www.osnews.com/thread?328110
I can search the body of all my messages in under 5 minutes, the headers in under 1. Gotta love that server-side indexing.


This is a huge amount of time. Searching messages on a web-based forum takes one second. I shouldn't expect an IMAP server to be slower. In both cases you're searching for a phrase in a list of messages. The IMAP server architecture makes the whole process slow. Five minutes is way too much to wait for.

Get a better IMAP server. For instance, Kolab uses Cyrus IMAP to store e-mail, contacts, and calendars in IMAP folders.


Thunderbird also has an extension that uses folders to store this information but I disagree with this practice. Folders should be used only for messages, and I would use one single large folder in the first place, each message would be labeled and indexed.

It does. Since when is calendars considered part of a "mail" protocol? Or even a "message" protocol"? ;)


Since when people started using email, calendar, address book and collaboration tools at the same time in the same application (this is why Zimbra, Lotus Notes and MS Outlook are so popular in companies). Needs are evolving and IMAP needs to extend its functionalities to meet these needs, especially for companies. Wed, 27 Aug 2008 05:53:00 GMT donotreply@osnews.com (Liquidator) Comments RE[3]: My rants with the IMAP protocol and IMAP servers http://www.osnews.com/thread?328176 http://www.osnews.com/thread?328176
"I can search the body of all my messages in under 5 minutes, the headers in under 1. Gotta love that server-side indexing.


This is a huge amount of time. Searching messages on a web-based forum takes one second. I shouldn't expect an IMAP server to be slower. In both cases you're searching for a phrase in a list of messages. The IMAP server architecture makes the whole process slow. Five minutes is way too much to wait for. "

Well, to each their own. I don't consider 5 minutes to search through several hundred thousand messages, and 1.5 GB of mixed text/binary data to be a long time. Maybe it's because I run my searches in the background while doing other things. Especially when you consider our IMAP server is also the webmail interface (squirrelmail), and is only a 1.6 GHz Opteron system with 4 GB of RAM, server just under 1500 users, about 500-ish logged in concurrently.

If we split that out into separate IMAP, web, auth, etc systems, and put in multiple CPUs and more RAM, the search time would drop. But, for now, we don't need it.

"Get a better IMAP server. For instance, Kolab uses Cyrus IMAP to store e-mail, contacts, and calendars in IMAP folders.


Thunderbird also has an extension that uses folders to store this information but I disagree with this practice. Folders should be used only for messages, and I would use one single large folder in the first place, each message would be labeled and indexed. "

So, do you want a groupware server that does everything, or do you want an IMAP server that just does mail? Make up your mind.

"It does. Since when is calendars considered part of a "mail" protocol? Or even a "message" protocol"? ;)


Since when people started using email, calendar, address book and collaboration tools at the same time in the same application (this is why Zimbra, Lotus Notes and MS Outlook are so popular in companies). Needs are evolving and IMAP needs to extend its functionalities to meet these needs, especially for companies. "

See above.

Zimbra, Lotus Notes, Exchange, and other similar groupware servers don't use IMAP for everything. In fact, they don't use IMAP for anything except access to mail folders from non-native clients. They each use their own custom protocol to communicate between the client and the server. You can't compare these to standard IMAP servers.

The Zimbra client, for example, communicates with the Zimbra server using SOAP calls for everything (mail, contacts, calendar, documents, notes, etc). However, they also provide an IMAP interface to the message store, an iCal interface to the calendars, a WebDAV interface to the documents storage, and an LDAP interface to the global addressbook.

IMAP is for accessing messages, nothing more. And it does that quite nicely. You just have to use the right server and client (especially the right server).

For examples of how to do IMAP wrong, just look at the FirstClass server, and MS Outlook. ;) Wed, 27 Aug 2008 15:54:00 GMT donotreply@osnews.com (phoenix) Comments RE[4]: My rants with the IMAP protocol and IMAP servers http://www.osnews.com/thread?328219 http://www.osnews.com/thread?328219
I don't consider 5 minutes to search through several hundred thousand messages, and 1.5 GB of mixed text/binary data to be a long time. Maybe it's because I run my searches in the background while doing other things.


I don't know any reaonable company that would accept it. My Gmail that I use for personal purpose does this job in half a second with the web-based interface.

Even if we want to limit IMAP strictly to email purpose, it still has the huge problem of lack of scalability. IMAP server are too slow when you reach a critical size. Wed, 27 Aug 2008 20:32:00 GMT donotreply@osnews.com (Liquidator) Comments RE[2]: My rants with the IMAP protocol and IMAP servers http://www.osnews.com/thread?328542 http://www.osnews.com/thread?328542
Get a better IMAP server. Remote search against Cyrus servers is very fast, even against my 1.5 GB work account. I can search the body of all my messages in under 5 minutes, the headers in under 1. Gotta love that server-side indexing.


Correction, get a better IMAP client. Doing searches from within Squirrelmail 1.5.0, it takes under 30 seconds to search through my 1.5 GB mailbox (body search). Using KMail 3.5.9, though, it takes just about 5 minutes to do the same search, against the same IMAP server. Not sure what's going on there. Fri, 29 Aug 2008 19:35:00 GMT donotreply@osnews.com (phoenix) Comments RE[5]: My rants with the IMAP protocol and IMAP servers http://www.osnews.com/thread?328543 http://www.osnews.com/thread?328543
"I don't consider 5 minutes to search through several hundred thousand messages, and 1.5 GB of mixed text/binary data to be a long time. Maybe it's because I run my searches in the background while doing other things.


I don't know any reaonable company that would accept it. My Gmail that I use for personal purpose does this job in half a second with the web-based interface. "

See my reply below. Seems my IMAP client was the bottleneck, not the IMAP server.

Even if we want to limit IMAP strictly to email purpose, it still has the huge problem of lack of scalability. IMAP server are too slow when you reach a critical size.


What's "critical size"? And what hardware is being used? Fri, 29 Aug 2008 19:36:00 GMT donotreply@osnews.com (phoenix) Comments