Setting up mail service on Ubuntu 8.04 "Hardy Heron"

Configuring Postfix, Dovecot IMAP, SASL, TLS, SpamAssassin, and ClamAV on Ubuntu "Hardy Heron" 8.04 LTS

October 31st, 2008

This is Part 1 of four parts:

Part 1: incoming mail with spam and virus blockers
Part 2: secure relaying for outgoing mail - authentication over SASL
Part 3: secure IMAP and SMTP relaying with TLS for Dovecot and Postfix
Part 4: the future - Apache JAMES?

Introduction - it's time to migrate our mail server

Our previous main server was running Suse Linux 10.1. That release has now reached end-of-life, and we're switching to Ubuntu Server 8.04 Long Term Support (LTS). We decided to go with this because we're using Ubuntu on our desktops, Ubuntu seems to have the momentum, and, frankly, Ubuntu's Debian-based package management is a joy, whereas Suse's package management system was barely usable. We also like the five-year support commitment for the Server LTS release.

One handy aspect of Hardy Heron is it comes with the VirtualBox virtual machine software. This means that I was able to install the Ubuntu Server release in a virtual machine to thoroughly test the entire system, including all the server configuration, before needing to do it on real servers. And after I finished creating these installation instructions, I was able to delete the virtual machine, recreate it, and test these instructions from start to finish, all quickly and easily on my desktop machine. My current desktop machine is running KUbuntu 8.04, but you could use VirtualBox on a Windows or OS X desktop also. VirtualBox is free and open source. I recommend using it to try out these instructions to see how everything works, before you run it on your real servers.

There are a few tricks for installing Ubuntu Server on VirtualBox and then accessing network services on the virtual server installation. We explain how to install Ubuntu Hardy Heron Server on VirtualBox

The email service migration is complex, because it involves incoming mail, outgoing mail, IMAP, and spam and anti-virus filtering. It would be nice if you could just install one package (apt-get install spamproof-antivirus-mta-and-imap perhaps?), and email is taken care of, but there's no such package at this point, and besides, there's so much variation in the way sites want to handle their email, it would be impossible to create a one-size-fits-all package like that.

We need Postfix to handle SMTP for both receiving mail, and for relaying outgoing mail. We need Dovecot for serving IMAP so users can read their email. And we need a suite of other packages to handle mail filtering to remove spam and viruses. And we need to do all of this in secure ways, including authentication for relaying, and mandatory TLS (encryption) for all authenticated connections. We've chosen Postfix as the MTA (but we'll explore another MTA in Part 4) and Dovecot as the IMAP server. We've selected SpamAssassin as our spam blocker, and ClamAV as our virus blocker. We've also selected Razor as our collaborative spam blocker; without it, SpamAssassin is much less effective. Finally, we use Amavisd-new to connect SpamAssassin and ClamAV to Postfix.

All this is under Ubuntu 8.04. We're using the package that come in the distribution, not packages we have compiled from source or gotten from some other repositories. This article assumes you are comfortable in administering Ubuntu and installing packages using apt-get.

There are many components involved. This part of the article (Part 1) deals with incoming email. A diagram helps to understand the flow of it all. This is a diagram of incoming mail, starting from receiving the email from the Internet, to delivering it to the users' mail queues.

Notes on Postfix and SMTPD processes

As you can see in the diagram, there are two Postfix SMTPDs running to receive emails. One takes them in from the Internet, and the other gets them back after filtering and delivers them.

Postfix uses the /etc/postfix/master.cf file to launch the collection of daemons which make up "Postfix". These daemons are all isolated process which perform small, well-defined tasks. The "isolated daemons" approach was developed as a reaction to the extremely long history of security problems with the monolithic process design of Sendmail.

master.cf runs the daemons located in the /usr/lib/postfix directory. See the master.cf documentation for details. We need to create two SMTPD processes so we'll create two lines. Each of these processes uses the same configuration file, namely, /etc/postfix/main.cf. To get different behavior for these two daemons, we add override arguments in the configuration line in master.cf. Doing so involves adding an indented (leading whitespace) line with an override flag:

    -o parameter=value

Note that there is no whitespace around the = sign, and there is some whitespace before the flag.

Step 1: configure an SMTPD to receive emails from the Internet

This SMTPD will listen on port 25 and accept any email routed to our domains. It will not deliver the email, but will instead route it directly to Amavisd for filtering. This SMTPD does only very simple filtering of rejecting emails that are not for our domains.

We'll put some settings in /etc/postfix/main.cf, settings which apply to all three of the Postfix SMTPD instances we will eventually create. The settings these will have in common are the mydomain and mydestination settings. The other settings, which are specific to the SMTPD instance, are specified using the overrides in the master.cf file.

We modify the default SMTPD server listening on port 25 to hand emails off to Amavisd. Find the default line:

smtp      inet  n       -       -       -       -       smtpd

which should be the first configuration line in master.cf. Comment the line out with a #.

We'll add a new line for our new SMTP which listens on Port 25 and forwards mail to Amavisd. We add an override to make it immediately forward all mail for our domains to Amavisd, which is running on 10024.

This line in master.cf creates an SMTP listener on port 25 to receive emails, and forwards it to a content filter named amavis-scan. Postfix is made up of a number of small single-purpose daemons, so the process that receives SMTP must actually forward it to another process that will send it out to Amavisd.

smtp         inet       n       -       n       -       5       smtpd
   -o content_filter=amavis-scan:[127.0.0.1]:10024

Here is the line which defines the outgoing process to send the received emails to Amavisd. This process talks to the smtpd over a Unix socket.

amavis-scan  unix       -       -       n       -       5       lmtp
             -o disable_dns_lookups=yes
             -o lmtp_send_xforward_command=yes
    -o lmtp_data_done_timeout=1200

We are using LMTP instead of SMTP as the protocol here. LMTP is the Local Mail Transport Protocol. It is the same as SMTP, but with a few more commands, to allow the server that received the email to pass more information on to Amavisd. In particular, it needs to pass on the IP address of the remote host, so Amavisd can pass that to SpamAssassin which will in turn check that IP address with Razor.

And this SMTPD listens on port 10026 to receive emails re-injected from Amavisd:

localhost:10026 inet    n       -       n       -       5       smtpd
                -o content_filter=
                -o mynetworks=127.0.0.0/8
                -o smtpd_authorized_xforward_hosts=127.0.0.0/8
                -o myhostname=localhost.chiralsoftware.com

We can also make some changes to main.cf for global settings. In our case, we specify our hostname and domains, and also the location of TLS certificates. The TLS certificates will be used later, when handling authenticated relaying.

Step 2: configure Amavisd to integrate spam blocking

Email today is unusable without spam blocking. Our system will use amavisd-new as an SMTP proxy. When Postfix receives a message, it will then forward it immediately through Amavisd. Amavisd will process the message by having it inspected SpamAssassin and ClamAV. If either of those two reject the message, the message will end up in a junk file. If the message passes, it will be sent back to Postfix for delivery.

Note that incoming messages, going through the secured relay, should not be filtered through Amavisd. The reason for this is that SpamAssassin looks at the SMTP client's IP address, and if the IP address is on blacklist, the mail will be blocked. If you have users who sometimes log in from overseas Internet cafes, it's quite likely that some of their IP addresses will be on these bad lists. We'll be using Razor to access these problem IP address lists. Some sites do actually run incoming messages through filters, to prevent virused machines from sending out messages, and to do whitelisting of email addresses. We might add this feature in a more advanced configuration.

Amavisd will act as an SMTP server, but will not be accessible to outside hosts. Amavisd itself will then be configured to call SpamAssassin and ClamAV. SpamAssassin itself is configured to work with the Razor collaborative blacklisting system. In our experience, spam filters must use a collaborative system like Razor to be effective.

Amavisd listens on Port 10024 by default. Use netstat -l -e -p again to verify that it is running on that port.

Configuring Amavisd is very easy. Usually, no changes are necessary, and it automatically detects the presence of Razor and ClamAV. In our setup, we do make one change, though. Amavisd normally re-injects mail on port 10025. Our remote users are used to using that port for authenticated relaying, so we want Amavisd to send it to 10026 instead. We add one line in /etc/amavisd/conf.d/50-user:

$forward_method = 'smtp:[127.0.0.1]:10026';

Amavisd automatically detects SpamAssassin if it is present. However, spam filtering must be turned on in Amavisd. Edit /etc/amavis/conf.d/15-content_filter_mode and uncomment the two lines to enable spam checking.

Other than that, we're done with Amavisd, and now we can move on to the filters Amavisd will use.

Step 3: Installing Razor

We'll install Razor before installing SpamAssassin.

Install the package:

apt-get install razor

Some initialization is needed for Razor. As root:

razor-admin -create

I ran it as root and it created various files in /root/.razor. That was not what I wanted to happen, so I delved in to it to find out what was causing this.

According to the documentation, the value of the razorhome configuration is set from the razor-agent.conf file. razor-agent.conf is found first in home/.razor, and if that does not exist, it is found in /etc/razor. I ran razor-admin with debugging enabled:

# razor-admin -d -debuglevel=9 -create

and indeed, razor-admin did not find razor-agent.conf in /root/.razor so, as expected, it looked in /etc/razor and found it. And the /etc/razor/razor-agent.conf does specify that razorhome is /etc/razor. So what is going on? What has happened is that razor-admin creates the directory /root/.razor and subsequently detects it and uses it as the razorhome. The solution to this is to remove the /root/.razor directory, and then run razor-admin with a -home option:

razor-admin -home=/etc/razor -create

Now Razor puts the right files in the right places and does not create a spurious /root/.razor directory. We need one more command to register our server with the Razor servers:

razor-admin -register

Unfortunately this command also fails. It detects the correct /etc/razor/razor-agent.conf file, which has a valid razorhome setting in it but it doesn't use that setting for some reason. Again, we use the -home option to fix this:

razor-admin -home=/etc/razor -register

Now Razor is ready, and we can install SpamAssassin.

Step 4: SpamAssassin

As always, installing the software is easy:

apt-get install spamassassin

It auto-detects that Razor is present but we do need to help it find out where Razor's configuration is kept. Edit /etc/spamassassin/local.cf and add:

razor_config /etc/razor/razor-agent.conf

I also changed the required_score setting from the default value of 5.0 to a more conservative value of 8.0.

Step 5: ClamAV

We'll expand on this section later.

Step 6: configure Postfix to do normal mail delivery for this domain

This one is easy. Postfix works almost out of the box. The default main.cf that comes with Ubuntu is very short. In our case, this machine doesn't yet have some of its DNS setting the way Postfix wants them, so we manually specify, towards the top of the file:

myhostname = chiralsoftware.net
mydomain = chiralsoftware.net

mydestination = $mydomain, chiralsoftware.com

Step 7: Test simple mail delivery

Restart Postfix and test out simple mail delivery. We'll send a message from the outside Internet to the SMTPD on Port 25. If the message is to a valid recipient, the message should be dropped into that user's mail queue file.

 telnet 70.38.78.37 10025
Trying 70.38.78.37...
Connected to 70.38.78.37.
Escape character is '^]'.
220 chiralsoftware.net ESMTP Postfix (Ubuntu)
MAIL FROM: <joe@yahoo.com>
250 2.1.0 Ok
RCPT TO: <bob@chiralsoftware.net>
250 2.1.5 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
This is a test
.
250 2.0.0 Ok: queued as 324B949C511
quit
        221 2.0.0 Bye

Local delivery worked. Look at the mail delivered in the user's spool, and notice that it went through three hops all on the local server:

Received: from localhost (localhost [127.0.0.1])
        by localhost.chiralsoftware.com (Postfix) with ESMTP id 6F3A049C56D
        for <hh@chiralsoftware.net>; Mon, 27 Oct 2008 18:19:16 -0400 (EDT)
Received: from chiralsoftware.net ([127.0.0.1])
        by localhost (local.chiralsoftware.com [127.0.0.1]) (amavisd-new, port 10024)
        with LMTP id WM5WBqyibplf for <bob@chiralsoftware.net>;
        Mon, 27 Oct 2008 18:19:16 -0400 (EDT)
Received: from me (chiralsoftware.net [205.134.230.202])
        by chiralsoftware.net (Postfix) with ESMTP id 7FB5E49C569
        for <bob@chiralsoftware.net>; Mon, 27 Oct 2008 18:18:58 -0400 (EDT)

We can see it has gone through Postfix, then Amavisd-new, and then Postfix again, for final delivery into the mail spool.

Test that spam filtering is active

All messages filtered through SpamAssassin should have headers like:

X-Spam-Flag: NO
 X-Spam-Score: 2.899
 X-Spam-Level: **

If there is no X-Spam-Flag header, SpamAssassin is not looking at it.

Incoming mail is complete

This completes the configuration necessary for incoming mail. Test it by sending messages through, and seeing that they end up in the user mail queues. Also verify that each message shows three hops in its headers, because it has gone from Postfix to Amavisd and back to Postfix.

For safety, test that neither the Amavisd, nor the re-injection Postfix, are accessible from the Internet. Verify this by going to ports 10026 and 10024, for the port settings described above. Both should fail.

Also, for safety, verify that the external Postfix running on port 25 is refusing to relay mail except to the domains it is configured for.

Now that incoming mail is working, we can go on to Part 2: secure relaying for outgoing mail - authentication over SASL. That section covers allowing users to send (relay) mail through the server, and fetch their mail with IMAP using Dovecot. Sending mail requires Postfix to authenticate users using Dovecot's SASL service.