Linked by Jon Jensen on Tue 26th Aug 2008 02:53 UTC, submitted by Ryan Masters
Internet & Networking The Book of IMAP: Building a Mail Server with Courier and Cyrus, by Peer Heinlein and Peer Hartleben, is a quality resource for any serious mail administrator. The approach taken is direct, but at the same time it's very expansive, setting this book apart from most others I have read. It's packed full of rich examples which are used to solidify the topic being covered. At several places the authors reach out to explain when the subject is addressing ambiguous or otherwise undocumented information which is to great advantage to the reader and worthy of recognition.
Order by: Score:
Liquidator
Member since:
2007-03-04

I use IMAP and Gmail and I think IMAP has to evolve...It shouldn't limit itself to just a simple remote folder and message manager.

1. It's slow. It wasn't designed for our gigabyte email accounts. When I have to sync my computer with my server, it synces one mail after the other. I have hundreds of thousands of email, it takes a huge amount of time. It should create a single zipped file of the headers and send it to my computer, it would be much faster than unzipped individual files. With numerous three-way TCP/IP connections, syncing with IMAP is just too slow.

2. Remote search is weak. Even with Dovecot. Try performing an advanced search and not only is it slow but often you don't get what you're searching because your IMAP server doesn't index attachments, sometimes it only searches headers and discards bodies.

3. No support for tags. While some MUA and webmails now migrated from folders to tags or labels, IMAP still uses folders, so for instance in Gmail I have "Projects" and "To do" tags, if a message has both labels, in IMAP it is physically duplicated.

4. No support for calendar/address book/RSS. I know IMAP is not a calendar server application but nowadays, many MUAs have calendar, RSS and address book integrated and it's essential to be able to sync a calendar, feeds and an address book without the need to set up an LDAP server.

5. Be in par with new needs. IMAP should offer all one expects from a mail protocol these days. I think when you use Gmail and IMAP, you shouldn't lose functionality, power, features and speed. It's time to dramatically upgrade the IMAP protocol, in the meantime, some people are migrating to web applications that have fast, synced, database-indexed emailing, calendar, RSS and address book.

Edited 2008-08-26 06:35 UTC

Reply Score: 2

Googol Member since:
2006-11-24

I am not sure whether you know what you are talking about.

Here's is one for you to try on YOUR machine: zip up a few hundred thousand mini-KB files and report back to us how long it took you. Then delete a random 50000 files and ad 70000 new one into that zip-file; then report back and tell us how long that took. Then do your math and let us know what will happen if the server has to do that to any number of people typically accessing their mail at a time... the server would explode ;)

Reply Score: 5

Liquidator Member since:
2007-03-04

I am not sure whether you know what you are talking about.


No comment.

Back to topic. I think the problem you're exposing is that IMAP servers work with *files*. Obviously, email servers would cripple if they had to zip individual small files. And they already cripple just retrieving individual messages. How many times do we get timeout error messages from Squirrelmail when we have too many messages in an IMAP folder? The solution is retrieving relevant messages from a *database*, storing them into a temporary file (single file), zipping this file and sending it to the MUA. This would solve our problem and servers would be happy.

Reply Score: 1

unoengborg Member since:
2005-07-06

There already are imap servers that use databases,
one of them is archiveopteryx http://www.archiveopteryx.org/

It uses Postgresql for backend storage, this means that you can make use of the extremly quick Postgresql full text search engine to search your mail.

Reply Score: 6

Liquidator Member since:
2007-03-04

The database is only used for backend storage. DBMail works this way for instance. But the client-server communication doesn't benefit from it.

Reply Score: 2

REM2000 Member since:
2006-07-25

i haven't done a start from scratch sync for a while, so i dunno if this is already the case.

However instead of the zip option perhaps being able to download multiple emails at the same time (parrallel download)

I do agree that IMAP and also Microsoft's implementation a la Exchange needs improving with the points you have listed. Email is a standard, many companies are using it more than telephone conversations. As commented before many people are reaching multi GB sizes, many people live in their email system, in some cases i wonder if the email system is becoming an operating system in itself.

Reply Score: 2

IMAP flags
by testerus on Tue 26th Aug 2008 11:30 UTC in reply to "My rants with the IMAP protocol and IMAP servers"
testerus Member since:
2005-07-06
phoenix Member since:
2005-07-11

1. It's slow. It wasn't designed for our gigabyte email accounts. When I have to sync my computer with my server, it syncs one mail after the other.


You're measuring the usage of a single IMAP client with a single IMAP server, and complaining that everything related to IMAP sucks? Yeah, that makes sense.

GMail uses IMAP over SSL, so you have an encryption/decryption process on top of the transport., slowing things down, depending on your CPU.

You don't specify which IMAP client you are using. Accessing GMail from KMail is nice and snappy, as KMail does multi-threaded message downloads/folder syncs.

Perhaps you should investigate other IMAP clients?

I have hundreds of thousands of email, it takes a huge amount of time. It should create a single zipped file of the headers and send it to my computer, it would be much faster than unzipped individual files. With numerous three-way TCP/IP connections, syncing with IMAP is just too slow.


I have 400 MB in my GMail account. Connecting to it from a brand new PC takes less than 5 sec to bring up the inbox (only new messages) and less than a minute to bring up the archive (just under 10,000 messages).

IOW, get a better IMAP client.

2. Remote search is weak. Even with Dovecot. Try performing an advanced search and not only is it slow but often you don't get what you're searching because your IMAP server doesn't index attachments, sometimes it only searches headers and discards bodies.


Get a better IMAP server. Remote search against Cyrus servers is very fast, even against my 1.5 GB work account. I can search the body of all my messages in under 5 minutes, the headers in under 1. Gotta love that server-side indexing.

Zimbra's Cyrus implementation in their OSS version is also quite speedy. And their Network edition is even faster.

3. No support for tags. While some MUA and webmails now migrated from folders to tags or labels, IMAP still uses folders, so for instance in Gmail I have "Projects" and "To do" tags, if a message has both labels, in IMAP it is physically duplicated.


Can't comment on this, as I don't use tagging in any of my IMAP clients.

4. No support for calendar/address book/RSS. I know IMAP is not a calendar server application but nowadays, many MUAs have calendar, RSS and address book integrated and it's essential to be able to sync a calendar, feeds and an address book without the need to set up an LDAP server.


Get a better IMAP server. For instance, Kolab uses Cyrus IMAP to store e-mail, contacts, and calendars in IMAP folders.

5. Be in par with new needs. IMAP should offer all one expects from a mail protocol these days.


It does. Since when is calendars considered part of a "mail" protocol? Or even a "message" protocol"? ;)

Edited 2008-08-26 23:20 UTC

Reply Score: 2

Liquidator Member since:
2007-03-04

I can search the body of all my messages in under 5 minutes, the headers in under 1. Gotta love that server-side indexing.


This is a huge amount of time. Searching messages on a web-based forum takes one second. I shouldn't expect an IMAP server to be slower. In both cases you're searching for a phrase in a list of messages. The IMAP server architecture makes the whole process slow. Five minutes is way too much to wait for.

Get a better IMAP server. For instance, Kolab uses Cyrus IMAP to store e-mail, contacts, and calendars in IMAP folders.


Thunderbird also has an extension that uses folders to store this information but I disagree with this practice. Folders should be used only for messages, and I would use one single large folder in the first place, each message would be labeled and indexed.

It does. Since when is calendars considered part of a "mail" protocol? Or even a "message" protocol"? ;)


Since when people started using email, calendar, address book and collaboration tools at the same time in the same application (this is why Zimbra, Lotus Notes and MS Outlook are so popular in companies). Needs are evolving and IMAP needs to extend its functionalities to meet these needs, especially for companies.

Reply Score: 1

phoenix Member since:
2005-07-11

"I can search the body of all my messages in under 5 minutes, the headers in under 1. Gotta love that server-side indexing.


This is a huge amount of time. Searching messages on a web-based forum takes one second. I shouldn't expect an IMAP server to be slower. In both cases you're searching for a phrase in a list of messages. The IMAP server architecture makes the whole process slow. Five minutes is way too much to wait for.
"

Well, to each their own. I don't consider 5 minutes to search through several hundred thousand messages, and 1.5 GB of mixed text/binary data to be a long time. Maybe it's because I run my searches in the background while doing other things. Especially when you consider our IMAP server is also the webmail interface (squirrelmail), and is only a 1.6 GHz Opteron system with 4 GB of RAM, server just under 1500 users, about 500-ish logged in concurrently.

If we split that out into separate IMAP, web, auth, etc systems, and put in multiple CPUs and more RAM, the search time would drop. But, for now, we don't need it.

"Get a better IMAP server. For instance, Kolab uses Cyrus IMAP to store e-mail, contacts, and calendars in IMAP folders.


Thunderbird also has an extension that uses folders to store this information but I disagree with this practice. Folders should be used only for messages, and I would use one single large folder in the first place, each message would be labeled and indexed.
"

So, do you want a groupware server that does everything, or do you want an IMAP server that just does mail? Make up your mind.

"It does. Since when is calendars considered part of a "mail" protocol? Or even a "message" protocol"? ;)


Since when people started using email, calendar, address book and collaboration tools at the same time in the same application (this is why Zimbra, Lotus Notes and MS Outlook are so popular in companies). Needs are evolving and IMAP needs to extend its functionalities to meet these needs, especially for companies.
"

See above.

Zimbra, Lotus Notes, Exchange, and other similar groupware servers don't use IMAP for everything. In fact, they don't use IMAP for anything except access to mail folders from non-native clients. They each use their own custom protocol to communicate between the client and the server. You can't compare these to standard IMAP servers.

The Zimbra client, for example, communicates with the Zimbra server using SOAP calls for everything (mail, contacts, calendar, documents, notes, etc). However, they also provide an IMAP interface to the message store, an iCal interface to the calendars, a WebDAV interface to the documents storage, and an LDAP interface to the global addressbook.

IMAP is for accessing messages, nothing more. And it does that quite nicely. You just have to use the right server and client (especially the right server).

For examples of how to do IMAP wrong, just look at the FirstClass server, and MS Outlook. ;)

Reply Score: 2

Liquidator Member since:
2007-03-04

I don't consider 5 minutes to search through several hundred thousand messages, and 1.5 GB of mixed text/binary data to be a long time. Maybe it's because I run my searches in the background while doing other things.


I don't know any reaonable company that would accept it. My Gmail that I use for personal purpose does this job in half a second with the web-based interface.

Even if we want to limit IMAP strictly to email purpose, it still has the huge problem of lack of scalability. IMAP server are too slow when you reach a critical size.

Reply Score: 2

phoenix Member since:
2005-07-11

"I don't consider 5 minutes to search through several hundred thousand messages, and 1.5 GB of mixed text/binary data to be a long time. Maybe it's because I run my searches in the background while doing other things.


I don't know any reaonable company that would accept it. My Gmail that I use for personal purpose does this job in half a second with the web-based interface.
"

See my reply below. Seems my IMAP client was the bottleneck, not the IMAP server.

Even if we want to limit IMAP strictly to email purpose, it still has the huge problem of lack of scalability. IMAP server are too slow when you reach a critical size.


What's "critical size"? And what hardware is being used?

Reply Score: 2

phoenix Member since:
2005-07-11

Get a better IMAP server. Remote search against Cyrus servers is very fast, even against my 1.5 GB work account. I can search the body of all my messages in under 5 minutes, the headers in under 1. Gotta love that server-side indexing.


Correction, get a better IMAP client. Doing searches from within Squirrelmail 1.5.0, it takes under 30 seconds to search through my 1.5 GB mailbox (body search). Using KMail 3.5.9, though, it takes just about 5 minutes to do the same search, against the same IMAP server. Not sure what's going on there.

Reply Score: 2