Configuring Exim and Courier IMAP under Debian GNU/Linux

On a Debian Woody or Sid box, the proceedure will be the following, to summarize:

In addition, I'm ignoring how mail is injected into Exim -- I use fetchmail to pull my Email down from TrekWeb, but you doubtless acquire your Email in other ways. It's also a forgone conclusion that I am assuming you're using Debian's default Mail Delivery Agent, Exim. Most of this discussion resvolves around setting up Exim related things, so if you're running something else and don't intend to switch to Exim the usefulness of this documention will likely be limited.

To get the ball rolling, you first need to make a few changes to your existing exim.conf file, usually in /etc/exim.

First, let's modify the transports section of exim.conf. The specific transport that we want to modify is local_delivery. When you're done, it should look something like what's below, from my exim.conf:

local_delivery:
  driver = appendfile
  group = mail
  mode = 0660
  mode_fail_narrower = false
  envelope_to_add = true
  return_path_add = true

  directory=${home}/Maildir
  maildir_format = true
  prefix = ""

The directory, maildir_format, and prefix options were added with the values above. This will enable side wide Maildir support. The ${home} variable is expanded out to the user's directory prior to a delivery. You can change the directory path to something else, but the default is the standard and recommended Maildir location.

If you'd like to be more conservative than that and just turn on per user Maildir, you can instead edit the address_directory transport and uncomment the maildir_format option, as shown below:

address_directory:
  driver = appendfile
  no_from_hack
  prefix = ""
  suffix = ""
  maildir_format

Then, edit the userforward director, adding directory_transport as shown below:

userforward:
  driver = forwardfile
  file_transport = address_file
  pipe_transport = address_pipe
  reply_transport = address_reply

  directory_transport = address_directory

  no_verify
  check_ancestor
  check_local_user
  file = .forward
  modemask = 002
  filter

directory transport should now point to the transport address_directory that you modified earlier to work with the Maildir format.

Furthermore, uncomment the filter option, as shown above, to enable user dot forward file (.foward) Exim filtering. This will enable your users the ability to filter their mail using Exim's powerful builtin filtering system, which works fairly well with Courier-IMAP, as shown below.

If you choose the latter method of enabling Maildir, then any user that desires her mail be stored in Maildir style needs to create a dot forward file like this:

jasonb@nebula:~$ cat .forward

/home/username/Maildir/

jasonb@nebula:~$

username should be the user's account name on the system. Exim will now deliver mail to this user using the directory_transport above. This simple dot forward file will need to be modified, however, if use of Exim's filters is desired.

Depending on how Exim is run on your system you'll either need to invoke the init.d script to restart it or restart (x)inetd so it sees your changes.

apt-get install the following packages. Pulling down courier-imap and courier-imap-ssl should fetch courier-base and the other related files for you.

nebula:/etc/courier# dpkg -l courier\*
||/ Name           Version
+++-==============-==============
ii  courier-authdaemon 0.37.3-2.1
ii  courier-base   0.37.3-7
ii  courier-imap   1.4.3-7
ii  courier-imap-ssl 1.4.3-3
ii  courier-ssl    0.37.3-3

(The latest Woody packages failed to function for me -- though this matter remains uninvestigated and apparently is now considered closed.)

It's important that your Exim configuration work correctly before proceeding further. Send yourself an email from another account somewhere on the Internet, or from another user on your own machine. (root works for this.) You might do the following:

nebula:/home/jasonb# mail -s "test message" jasonb<ENTER>
test<ENTER>
^D
Cc:<ENTER>

(Use the Control-D sequence, ^D, don't actually type it.)

The purpose of this exercise is two fold. First, you'll find out if Exim is still working properly. Second, and equally as important, Exim will create the Maildir hierarchy in your home directory, ~/Maildir, which Courier IMAP craves and requires to function.

It's necessary for each user to have this directory before Courier can successfully open a user's Inbox. With that in mind, there are a few ways you can ensure that this directory exists. For future users, you can modify your /etc/skel directory, where all files destined to be placed in future user accounts live, by adding a Maildir there. Thusly as root:

maildirmake /etc/skel/Maildir

All future users will now have a ~/Maildir with the proper permissions and subdirectories.

That still leaves existing users, though. You can mail everyone on the system about the new IMAP setup and in doing so, Exim will create the very Maildir structure you seek. Or you can write a script to add a Maildir for each existing user with the maildirmake command like we did above for /etc/skel. It's up to you.

You cannot proceed with the next step successfully, in any case, without at least building a Maildir for yourself. So if you didn't send a test message, run maildirmake Maildir from the root of your home directory as your user.

Curtis Nelson explained to me via email that for console applications, like Mutt, you'll want to change your /etc/login.defs file like so:

QMAIL_DIR    Maildir/
#MAIL_DIR              (comment this out)
MAIL_FILE    Maildir/

And additionally, if you modify your /etc/pam.d/login like the following, you'll receive new mail notifications.

session optional pam_mail.so standard noenv dir=~/Maildir

Ernest Turro-Bassols adds that the above should be added to your /etc/pam.d/ssh file if you want new mail notification when you connect to your box via SSH.

Before going through the process of configuring IMAP clients, let's verify that your setup does indeed work. The default Courier-IMAP configuration should work right out of the box. Telnet to your IMAP server as shown below and issue the commands show and verify the server's response. (imap2 is port 143 if you're curious.)

jasonb@nebula:~$ telnet 127.0.0.1 imap2
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
* OK Courier-IMAP ready.
  Copyright 1998-2002 Double Precision, Inc.
  See COPYING for distribution information.

AB LOGIN "username" "password"
AB OK LOGIN Ok.

BC SELECT "Inbox"
* FLAGS (\Draft \Answered ... \Recent)
* OK [PERMANENTFLAGS (\Draft \Answered ... \Seen)] Limited
* 23 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1026858715] Ok
BC OK [READ-WRITE] Ok

ZZZZ LOGOUT
* BYE Courier-IMAP server shutting down
ZZZZ OK LOGOUT completed

Connection closed by foreign host.

I seperated each command and server response portion by a couple hard returns to make things more clear. Make sure you substitute your username and password for the dummy values above, and note that the AB, BC, et al. are part of the IMAP protocol and are required, though any sequence of two letters will do. (Also, be amused that the actual logout command is indeed ZZZZ LOGOUT -- Someone had a sense of humor.)

In OutLook Express v5.5™, select the Tools menu, then select Accounts. Now, select the Mail tab and click on Add. Select Mail... and go through the wizard until you come to the E-mail Server Names section. Here you'll want to enter the IP or hostname of your IMAP server and select IMAP as the server type. For Internet Mail Logon, leave the Secure Password Authentication box unchecked as you will not be using this. Finish, then select your new account from the list under the Mail tab and select Properties. Select the IMAP tab and make sure you enter "Inbox" as your Root folder path without the double quotes. This is important so the folder hierarchy shows up correctly. If you ever modify, add or delete folders from another IMAP client, you'll need to select Reset List for your account by right clicking on it in OutLook Express' main window.

In Kmail v1.4.5™ and above, choose Settings and Configure KMail and then select the Network icon. Switch to the Receiving tab and then Add... to add a new account. Select the IMAP radio button. Now, enter a name for this account, a username, password, and your IMAP server's IP or hostname. Next, enter "INBOX", without the double quotes, as your Prefix to folders. This ensures that your folder hierarchy is displayed correctly and you can add, modify, and remove folders without errors.

In Mozilla 1.0™'s Email client, select Edit and then Mail & Newsgroups Account Settings. From there, Add Account..., choose Email account, then follow the wizard through until you get to the Server Information window. Select IMAP and enter your IMAP server's IP or hostname. Finish. Now, select the Server Settings entry for your newly created account and click on the Advanced... button. For your IMAP server directory enter "INBOX/" without the quotes. This will ensure your folder hierarchy is displayed properly. Hit OK to complete setting up your new IMAP account.

As you read through the above, you should have picked up on the "INBOX" theme. While an unofficial extension to the IMAP protocol announces this prefix, the above three clients don't bother to use it. So you need to enter it manually. It's possible other clients not discussed here will pick up on this automagically, but if not search out the prefix or server directory option for your favorite client and add some variant of inbox there. The client might want it in a specific case or with a trailing slash, or it might convert it to its desired scheme after you save your changes.

No discussion would be complete without relevant security information. For Courier-IMAP on Debian, it's literally as simple as running:

apt-get install courier-imap-ssl

The installation script will generate an X.509 certificate suitable for usage by IMAP clients. At that point, you can configure your mail client to use IMAP over SSL on default port 993. It's important to note that the default certificate generated is not signed by any official key authority, so it will fail any authenticity tests, but most clients will allow you to ignore that warning.

In OutLook Express v5.5™, select your IMAP account from your folder list and right click, select Properties. Then, go to the Advanced tab and check This server requires a secure connection (SSL) for your Incoming mail server.

In Kmail v1.4.5™ and above, choose Settings and Configure KMail and then select the Network icon. Switch to the Receiving tab and then select your IMAP account and choose Modify. From there, select the Security tab and click See What the Server Supports. KMail should respond with two warnings about the invalidity of your X.509 certificate. Accept the certificate anyway and tell it to use it forever, or for this session. Your choice. KMail should then finish determining what the server supports and should now have Use SSL for secure mail download selected by default. You're done, hit OK.

In Mozilla 1.0™'s Email client, select Edit and then Mail & Newsgroups Account Settings. From there, select your IMAP account and select the Server Settings option immediately beneath it. Check the Use secure connection (SSL) option. Mozilla will warn you your X.509 certificate is signed by an unknown authority. You can allow Mozilla to remember it and bother you no further.

For a detail discussion of Exim's filter system, you should check out Exim's documentation itself. For just the short of it, read on.

In order for Exim to recognize a dot forward as having Exim filter rules, the first non white space entry must be:

# Exim filter

Case and anything following it on the same line are ignored.

Next, you'll likely want some rules in your Exim filter dot forward file. Using a simple if then elif else endif construct you can perform various tests against each incoming email before delivery (or discarding) it to a specified location. You can perform evaluations against any existing entry in an email's header, like so:

# Match any email who's To: header contains "exim"
# and save it to .dir1
if $h_to: contains "exim" then save Maildir/.dir1/

# Match email with a From: header that's
# exactly "not@wanted.com" and save it to .SPAM
elif $h_from: is "not@wanted.com" then save Maildir/.SPAM/

endif

To access the email header of your choice, append $header_, or as abbreviated above, just $h_, to the full name of an email header, following by a colon (:). The keywords is and contains are self explanatory. The save keyword, when succeeded by a path that ends with a forward slash (/), will deliver the email being evaluated in Maildir format. As such, the trailing slash is crucial.

In Courier's IMAP hierarchy, directories beneath the root are dot directories. In addition, all subdirectories are denoted by periods, not additional forward slashes. So, lists/Debian/User/ is actually .Lists.Debian.User/ on disk and should be referred to in Exim filters as "save Maildir/.Lists.Debian.User/" for things to be saved the way you expect.

A successful delivery ought to look like this (but on three lines instead of more):

 2002-07-25 01:53:14 17XbYo-000365-00 <=
prism2@example.org H=localhost [127.0.0.1] P=esmtp S=5880
id=3D3F91FC.6090902@example.org

2002-07-25 01:53:15 17XbYo-000365-00 =>;
/home/jasonb/Maildir/.mailinglists.HostAP/ <jasonb@localhost>
D=userforward T=address_directory

2002-07-25 01:53:15 17XbYo-000365-00 Completed 

My own personal working Exim filter file looks like this:

# Exim filter

# Handle mailing lists

if $h_to: contains "leaplist"
  then save Maildir/.mailinglists.leap/

elif $h_to: contains "debian-kde"
  then save Maildir/.mailinglists.Debian.KDE/

elif $h_from: contains "ebay.com"
  then save Maildir/.Ebay/

elif $h_delivered-to: contains "hostap"
  then save Maildir/.mailinglists.HostAP/

elif $h_to: contains "LINUX-L" or $h_to: contains "linux-l"
  then save Maildir/.mailinglists.LUG/

elif $h_to: contains "linux@flux"
  then save Maildir/.mailinglists.FLUX/

elif $h_from: contains "half.com"
  then save Maildir/.mailinglists.books/

elif $h_subject: contains "StuBex.com"
  then save Maildir/.mailinglists.books/

endif

# If you didn't enable site wide Maildir
# your users will need to have this catch all
# entry, or it'll end up in MBox format.
#
# save Maildir/

# Done

I've since revised my Exim filters to use the List-Post and X-Mailing-List headers to deal with the ocassional Emails that are CC'd to lists I'm subscribed to. Better ideas are welcome.

It's possible your Exim retry.lockfile may get adopted by root, as alluded to in this thread. This error is symptomatic of such an adoption:

2002-07-24 12:40:52 17XP8Z-0005i2-00 failed to open database
lock file /var/spool/exim/db/retry.lockfile:
Permission denied (euid=8 egid=8)

Executing a

chown mail:mail /var/spool/exim/db/*

should resolve the issue -- it did for me.

Another error you may encounter is symptomatic of forgetting to enable user forward file filters in /etc/exim/exim.conf:

 2002-07-24 12:52:17 17XOsd-0005fo-00 == user@localhost
D=userforward defer (-11): error in forward file
(filtering not enabled): missing or malformed
local part (expected word or "<") in "... filter rule ..." 

Though the error mentions several problems, the actual issue is, as it states, filtering isn't enabled for user forward files. Reread the section above on enabling user forward file filters.

It's also possible you might get this error, which results from a file by the same name already existing in your Maildir:

 2002-07-28 07:36:13 17YmLM-0002bu-00 ==
/home/jasonb/Maildir/.mailinglists.EX/ <jasonb@localhost>
T=address_directory defer (20):
Not a directory: stat() error for
/home/jasonb/Maildir/.mailinglists.EX/: Not a directory 

It's often the result of forgetting to include the trailing front slash (/) in an Exim filter rule and then later adding it. In the meantime, Exim will happily save the mail you're directing there in MBox format instead of Maildir format. Later when you fix your filter, if you don't remember to move the MBox dot file out of the way, your mail will get frozen by Exim.

If you're getting disconnected from Courier IMAP without any obvious error message, you might've fallen victim to your Maildir being violated. If you run `tcpdump` against your session, you might see something like the following output I obtained:

* OK [CAPABILITY IMAP4rev1 UIDPLUS ... ] Courier-IMAP ready. \
  Copyright 1998-2003 Double Precision, Inc. \
  See COPYING for distribution information..
0 CAPABILITY.
...
0 OK CAPABILITY completed.
1 LOGIN "jasonb" "[protected]".
1 OK LOGIN Ok..
...
14 LIST "" "INBOX.mailinglists.leap".
* LIST (\Unmarked \HasChildren) "." "INBOX.mailinglists.leap".
14 OK LIST completed..
15 STATUS "INBOX.mailinglists.leap" (UNSEEN).
* BYE [ALERT] Fatal error: No such file or directory.

At that point, the IMAP client is disconnected. However, that's not very helpful, so you may wish to run an `strace` on the process to investigate further.

nebula:~# ps auxw | grep courier
...
root 242  0.0  0.1  1476  516 ? S 00:54 0:00
/usr/sbin/couriertcpd -address=0 -stderrlogger=/usr/sbin/courierlogger
-maxprocs=40 -maxperip=4 -pid=/var/run/courier/imapd.pid -nodnslookup
-noidentlookup 143 /usr/lib/courier/courier/imaplogin
/usr/lib/courier/authlib/authdaemon /usr/bin/imapd Maildir
...
nebula:~# strace -F -f -eopen -p 242
... magic ...
[pid   807] open(".mailinglists.leap/tmp",
O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = -1 ENOENT (No such file or
directory)
...
Process 807 detached

Now, it becomes clear. Somehow, one of the necessary Maildir directories seems to have disappeared. Recreating this directory solves the problem, though not necessarily the mystery of how it disappeared.

jasonb@nebula:~/Maildir$ mkdir tmp
jasonb@nebula:~/Maildir$ chmod u=rwx,g=s,o= tmp/

A big thanks to David Pashley. Without his patience and expertice in this matter I could not have written this document or set it up locally. Thanks!

Thanks to William Luecke, Robert William Hutton, Curtis Nelson, Ernest Turro-Bassols (and others) as well!