tutorials and ruby scripts

Read my tutorials


Tutorial: mailserver

This tutorial is about setting up a mailserver. It is based on my own efforts to not use gmail any longer.
I want to keep my own emails and I want to keep my own backups.

Running your own mail server is pain. This is something you have to monitor all the time. And as long as your server is not encrypted (and even if) you should use a client based enryption for your email client.

It is also a good idea to separate this from your other services. I am using a small 128 MB vps that is only running the mail server. It is fine for 5 users. If you want to run additional services like clamav and if you have more than 5 users you should use 512 MB of RAM.

This post will be updated once a week. This topic is quite huge so I will need some iterations to complete this tutorial. I also want to include all feedback to ensure that this tutorial is up to date.

Because email servers can have a lot of features like:

  • marking spam
  • graylisting
  • virus scanning
  • virtual mappings
  • etc

I will mark every step that is just adding a feature as OPTIONAL.

So let’s start with the preparations:

**1. Setup your DNS:**

Create an A record that is pointing to your vps (which should run the mailserver).

mailserver A 1800
mailserver 2500:f5f5:25::b25f:2525 AAAA 1800

I use a service oriented nameing shema so in my case: “mailserver.domain.com”. This name is quite important because it is used in a lot of different places.

You can add additional cnames to ensure that all the mail clients find the correct ips:

pop3 mailserver.domain.com. CNAME 1800
pop mailserver.domain.com. CNAME 1800
imap mailserver.domain.com. CNAME 1800
smtp mailserver.domain.com. CNAME 1800

And the AAAA records too if you want to support IPv6.

Create a MX record for your domain and subdomains

@ mailserver.domain.com. MX 10 3600

“@” is an alias for you domain. So all email for your domain should be sent to “mailserver.domain.com”

You have to create records for your subdomains too:

mysubdomain mailserver.mydomain.com. MX 10 3600

And to ensure that SPF gets more support add this TXT DNS entry too:

  @		IN	TXT	"v=spf1 mx -all"

This adds the additional security that you say that only your MX entries are allowed to send emails for your domains. Quite obvious but you can add other ips too:

  @		IN	TXT	"v=spf1 mx ip4: a:mail.company.com -all"

This states that all MX servers, the IP and the mail.company.com are allowed to send emails for your domains.

Why? Because sometimes (e.g. for forums/mailing lists) an external company is sending emails for your domains. This is a way to approve them.

2. Setup your rDNS:
Go to your vps control panel and add the DNS record “mailserver.domain.com” to your IP address

3. Setup your mailname in /etc

sudo nano /etc/mailname
Add "mailserver.mydomain.com"

4. Setup your iptables rules:

 # allow SMTP
 iptables -A INPUT -i $device -m state --state NEW -p tcp --dport 25 -j ACCEPT
 ip6tables -A INPUT -i $device -p tcp --dport 25 -j ACCEPT

 # allow SMTPS
 iptables -A INPUT -i $device -m state --state NEW -p tcp --dport 465 -j ACCEPT
 iptables -A INPUT -i $device -m state --state NEW -p tcp --dport 587 -j ACCEPT

 ip6tables -A INPUT -i $device -p tcp --dport 465 -j ACCEPT
 ip6tables -A INPUT -i $device -p tcp --dport 587 -j ACCEPT

 # allow POP3
 iptables -A INPUT -i $device -m state --state NEW -p tcp --dport 110 -j ACCEPT
 ip6tables -A INPUT -i $device -p tcp --dport 110 -j ACCEPT

 # allow POP3S
 iptables -A INPUT -i $device -m state --state NEW -p tcp --dport 995 -j ACCEPT
 ip6tables -A INPUT -i $device -p tcp --dport 995 -j ACCEPT

 # allow IMAP
 iptables -A INPUT -i $device -m state --state NEW -p tcp --dport 143 -j ACCEPT
 ip6tables -A INPUT -i $device -p tcp --dport 143 -j ACCEPT

 # allow IMAPS
 iptables -A INPUT -i $device -m state --state NEW -p tcp --dport 993 -j ACCEPT
 ip6tables -A INPUT -i $device -p tcp --dport 993 -j ACCEPT

5. Setup your mail server
Installation is really simple because of the great package: dovecot-postfix

sudo apt-get install dovecot-postfix

After using some other mail daemons for a while I do prefer the postfix/dovecot combo.

Postfix is mail daemon like sendmail but whith a real nice pipe framework. It is really easy to tunnel email through different modules. Dovecot is a daemon that provides pop3 and imap access to the mail accounts.

6. Config dovecot
Dovecot is providing access to your emails via pop/imap.
Only file to edit is /etc/dovecot/conf.d/10-auth.conf

nano /etc/dovecot/conf.d/10-auth.conf

Remove the # of the line “disable_plaintext_auth = yes”

7. Config postfix
Edit the file /etc/postfix/main.cf

nano /etc/postfix/main.cf

Things to edit:

mydomain = domain.com
myorigin = domain.com
myhostname = mailserver.domain.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
local_recipient_maps = $alias_maps
mydestination = domain.com, mailserver.domain.com, subdomain.domain.com, localhost

smtpd_recipient_restrictions = 
  reject_rbl_client zen.spamhaus.org, 
  reject_rbl_client bl.spamcop.org

smtpd_data_restrictions =

So what I am doning here?

I define the domain and origin of the mail server. He should handle the domain “domain.com”.

I define an alias map to map different email accounts to different linux users.

And I define all allowed destinations - including all subdomains. And of course “localhost” for all my scripts.

All mail for a different domain will be rejected.

“smtpd_recipient_restrictions” is a list of filters to ensure that we did not get spammed.

“reject_rbl_client” is a referrer to one of the spam lists provided by different groups. I do like spamhaus and spamcop.

“check_policy_service” is used for my favorit greylister. Greylister do something very bad. They dismiss emails - out of the fact that real and good mail servers will try again. A lot of spammers don’t have time to wait for any retries.

I know that this is … still in discussion … but for a private mail server it just saves a lot of time.

Another time saver is local_recipient_maps. So every email like “[email protected]” will be rejected if the mail address is not within the alias map. You do only receive emails to mailboxes you added.

Edit the file /etc/postfix/master.cf

nano  /etc/postfix/master.cf 

Things to edit:

submission inet n       -       -       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

dovecot   unix  -       n       n       -       -       pipe
  flags=DRhu argv=/usr/lib/dovecot/deliver -f ${sender} -d ${recipient}

Be carefull … spaces etc. do count!

So what did I change?

I have added an output pipe for spam assassin and one for dovecot.

Postfix is recieving all mails and has to forward them to dovecot. Additionally a content_filter is set (defined at the end of the file) to ensure that the mails are sent to spamassassin and afterwards to dovecot.

8. Config aliases

I do prefer aliases instead of virtual mappings. I don’t want to run a MySQL server to choose who is getting what email. There are a lot of tutorial on how to use virtual mappings. I stick to simple config files.

There are some DDOS attacks too that aim at a high load of MySQL querries too.

Now edit the file /etc/aliases

nano /etc/aliases


# Mail aliases for sendmail
# You must run newaliases(1) after making changes to this file.

# Required aliases
postmaster:	wlanboy
MAILER-DAEMON:	postmaster

# Common aliases
abuse:		postmaster
spam:		postmaster

# Other aliases
webmaster:      wlanboy
contact:        wlanboy
root:           wlanboy
user1:          user1
wlanboy:        wlanboy

Double check that you do not map circles like: postmaster -> spam -> wlanboy -> admin -> admin -> postmaster.

You wont have any chance to see the cause in the logfiles.

On the left side are email addresses like “[email protected]”.

On the rigth side there are linux users which will receive the emails.

Afterwards you have to run “newaliases” to generate the alias map file.

sudo newaliases

To add a new user just type:

sudo useradd -m -s /bin/false [username]
passwd [username]

This ensures that this user can only login into your mail server and not use any other services like ssh/scp/rsync.

9. OPTIONAL: Install postgrey

sudo apt-get install postgrey
sudo nano /etc/default/postgrey
POSTGREY_OPTS="--inet= --delay=55"

The OPTS I add are:

  • listening to localhost only
  • add a delay of 55 seconds

Afterwards you have to edit the file /etc/postfix/main.cf

nano /etc/postfix/main.cf
smtpd_recipient_restrictions = 
  reject_rbl_client zen.spamhaus.org, 
  reject_rbl_client bl.spamcop.org,
  check_policy_service inet:

Add the check_policy_service inet: to the smtpd_recipient_restrictions.

10. OPTIONAL: Install spam assassine

Enhance postfix configuration:

nano /etc/postfix/master.cf

Things to edit:

smtp      inet  n       -       -       -       -       smtpd
  -o content_filter=spamassassin

submission [.....]
dovecot [.....]

spamassassin unix -     n       n       -       -       pipe
        user=spamd argv=/usr/bin/spamc -f -e    
        /usr/sbin/sendmail -oi -f ${sender} ${recipient}

And now install spam assassine

sudo apt-get install libnet-dns-perl pyzor razor libdigest-sha-perl libencode-detect-perl libdbi-perl libgeo-ipfree-perl libnet-ident-perl 
sudo apt-get install spamassassin
sudo adduser --system --no-create-home spamd

Afterwards you have to activate it:

sudo nano /etc/default/spamassassin
change to: ENABLED=1
sudo service spamassassin restart

Update spam asssasin rules:

cd /etc/spamassassin/
wget http://yerp.org/rules/GPG.KEY
sa-update --import GPG.KEY
sa-update --gpgkey 6C6191E3 --channel sought.rules.yerp.org

sa-update -D -v

11. Install fail2ban

sudo apt-get install fail2ban

You have to configure the services fail2ban has to check:

sudo nano /etc/fail2ban/jail.conf

Things to edit:

bantime  = 3600
maxretry = 2

action = %(action_mw)s

enabled  = true

enabled  = true
port     = smtp,ssmtp
filter   = postfix
logpath  = /var/log/mail.log

enabled  = true
port     = smtp,ssmtp,imap2,imap3,imaps,pop3,pop3s
filter   = sasl
# You might consider monitoring /var/log/mail.warn instead if you are
# running postfix since it would provide the same log lines at the
# "warn" level but overall at the smaller filesize.
logpath  = /var/log/mail.log

enabled = true
port    = smtp,ssmtp,imap2,imap3,imaps,pop3,pop3s
filter  = dovecot
logpath = /var/log/mail.log

What did I do?

Set the bantime to 1 hour and the number of retries before ban to 2.

And enabled the observation of pam-generic, postfix, sasl and dovecot.

So all mail related login actions are checked.

12. OPTIONAL: Own SSL configuration

The package is creating self signed certificates. So if you want to change them because you want to use official ssl certs edit following lines:

  • /etc/postfix/main.cf
  • /etc/dovecot/conf.d/10-ssl.conf
  • /etc/dovecot/conf.d/01-mail-stack-delivery.conf
/etc/postfix/main.cf:smtpd_tls_cert_file = /etc/ssl/certs/ssl-mail.pem
/etc/postfix/main.cf:smtpd_tls_key_file = /etc/ssl/private/ssl-mail.key

/etc/dovecot/conf.d/10-ssl.conf:ssl_cert = 

13. OPTIONAL: Set rate limits

If your mail server is used by yourself … you do not need to limit the number of emails a user can send.

Edit /etc/postfix/main.cf

sudo nano /etc/postfix/main.cf

And add these lines at the end of file:

smtpd_client_event_limit_exceptions = $mynetworks
#Clients that are excluded from connection count
anvil_rate_time_unit = 60s 
#The time unit over which client connection rates and other rates are calculated. 
anvil_status_update_time = 120s 
#How frequently the server logs peak usage information. 
#The maximal number of message delivery requests that any client is allowed to make to this service per time unit.

So each client - not connected through $mynetworks - is only able to send 5 emails per 60 seconds.


Well ….

DomainKeys Identified Mail (DKIM) is a method for E-mail authentication, allowing a person who receives email to verify that the message actually comes from the domain that it claims to have come from. The need for this type of authentication arises because spam often has forged headers.

So your mail server can sign your emails to ensure that someone can check if the emails are from your approved mail servers.

Installation is quite easy:

sudo apt-get install opendkim opendkim-tools

For 12.04 you have to use backports:

sudo apt-get install opendkim/precise-backports
sudo apt-get install opendkim-tools/precise-backports

Configuration is done on two files:


Things you have to change:

nano /etc/opendkim.conf
UserID                 105 # 'id postfix' in your shell
Domain                 domain.com
KeyFile                 /etc/mail/dkim.key
nano /etc/default/opendkim
SOCKET="inet:54321" # listen on all interfaces on port 54321
#Don't forget to allow this port on iptables

Now we have to tell postfix to use this service:

nano /etc/postfix/main.cf
milter_default_action = accept
milter_protocol = 2
smtpd_milters = inet:localhost:8891
non_smtpd_milters = inet:localhost:8891

To generate the key run following command:

cd /etc/mail
sudo opendkim-genkey -t -s mail -d domain.com
cp mail.private /etc/mail/dkim.key

The DNS TXT entry should be a copy&paste of mail.txt

mail._domainkey.domain.com. IN TXT "v=DKIM1; g=*; k=rsa; p=openssl_public_key";

15. OPTIONAL: Add backup MX

First of all you have to add an additional MX record with a higher priority:

@ mailserver.domain.com. MX 10 3600
@ backupmailserver.domain.com. MX 20 3600

Everyone is first trying to send the email to mailserver.domain.com, if it is not reachable backupmailserver.domain.com is used.

The higher the priority the lower the chance that someone is using the MX server.

Next change of the backup mail server is the main.cf:

relay_domains = $mydestination, hash:/etc/postfix/relay_domains
transport_maps = hash:/etc/postfix/relay_transport

Now we have to define the relay domains:

nano /etc/postfix/relay_domains


domain1.com anything
domain2.com anything
domain3.com anything

A backup MX can be responsible for more than one domain.

Now we have to define what the backup mail server should do with an incoming email:

nano /etc/postfix/relay_transport


domain1.com relay:mailserver.domain1.com
domain2.com relay:mailserver.domain2.com
domaint3.com relay:mailserver.domain3.com

Just forward them to the real mail servers.

Last step is to map both files:

cd /etc/postfix
postmap relay_domains
postmap relay_transport

If you want you can add a time to live for the emails too:

maximal_queue_lifetime = 60d 

So the backup server will store the mails for 60 days. Hopefully your main mail server will not be offline for more than 60 days.

16. Restart you vps



We now have a smtp/pop3/imap server that uses graylisting, spam assassine and a white list of mail addresses to ensure that you only receive mails you want.

Additionally fail2ban bans everyone trying to get access to your mail server.

Postfix and dovecot are by default supporting IPv6. You only have to add the AAAA records to ensure you mail server is accessable via IPv6.

You can even decide to drop the iptables/ip6tables rules for SMTP/POP3/IMAP to ensure every client is only using SSL secured connections.

If you need a GUI for this mail server:

Use a second vps with webserver and php and install roundcube.

**Additional notes:**

Yup you are right there is not a fancy clicky GUI to add new mailboxes. But I like the idea to keep things simple.

Adding a user to a linux system (one without a console access) is dead simple. Adding an alias for him too.

This is a private server - so you will not add new users every minute.

A lot of things a file based so the whole system is not using a lot of resources. A real low end mail server running on a 128 MB vps:

             total       used       free     shared    buffers     cached
Mem:        131072      82560      48512          0          0      28564
-/+ buffers/cache:      53996      77076
Swap:       131072      15808     115264

This is the state without any active connections. Dovecot is using some RAM for each logged-in user.

This tutorial is quite long and as you can see most of the steps is about securing your mail server. Mail servers are still targets of quite a lot of attacks. These attacks are simple and don’t need a lot of time of traffic. So an easy target. The ports cannot be changed too so you know that port 25 is listening.

So a lot of arguments to run your mail server on another vps.