IPsec Between Openswan and Windows Using Certificates

In this post we configure Openswan and Windows 7 (or Vista) to bring up an IPsec + L2TP tunnel. One point of difference is that we focus on using certificates to facilitate secure IPsec connectivity. Put another way, we use RSA certs on phase1, rather than a pre-shared key. Configuring and maintaining this properly is not easy.

We will also look at how the Windows Vista and Windows 7 clients are configured with certificates (spoiler: it's the same way for both clients).

I'm a bit of an Openswan fan. I don't exactly know why, since strongSwan seems to be more feature rich. Maybe I defer to Openswan because it's easily installed through any Linux distro's package manager. Or perhaps because I started muddling around with it back in 2005. Anyway, a lot of what is written here will carry over to strongSwan.

Openswan seems to lack features that strongSwan offers; the ability to configure lifetime bytes (lifebytes) was one Openswan limitation I most recently bumped up against.

If there's anything in all this that isn't clear, then I'm happy to field comments or questions.

If you're wondering about the Oklahoma references later, it's because yesterday I watched the 1940 adaptation of The Grapes of Wrath.

Server Version

# cat /etc/debian_version
7.11

Files

/etc/ipsec.conf
/etc/ssl/openssl.cnf
/etc/xl2tpd/xl2tpd.conf
/etc/ppp/options.l2tpd.lns
/etc/ppp/chap-secrets

Openswan

Simply take the example configuration and make amendments in a few places. Insert the contents of this file into the /etc/ipsec.config file. You can just cat the file appending it to ipsec.conf. For example:

cat l2tp-cert.conf >> /etc/ipsec.conf

The examples directory contains lots of handy files:

# ls -la /etc/ipsec.d/examples/
total 44
drwxr-xr-x  2 root root 4096 Mar  9 20:12 .
drwxr-xr-x 10 root root 4096 Mar  9 20:12 ..
-rw-r--r--  1 root root 1659 May 27  2012 hub-spoke.conf
-rw-r--r--  1 root root 1017 May 27  2012 ipv6.conf
-rw-r--r--  1 root root 1736 May 27  2012 l2tp-cert.conf
-rw-r--r--  1 root root 1825 May 27  2012 l2tp-psk.conf
-rw-r--r--  1 root root 1156 May 27  2012 linux-linux.conf
-rw-r--r--  1 root root 1580 May 27  2012 mast-l2tp-psk.conf
-rw-r--r--  1 root root  235 May 27  2012 oe-exclude-dns.conf
-rw-r--r--  1 root root 1694 May 27  2012 sysctl.conf
-rw-r--r--  1 root root  664 May 27  2012 xauth.conf

 The following example config has all the helpful comments snipped out for brevity:

# cat /etc/ipsec.conf
version 2.0     # conforms to second version of ipsec.conf specification
# basic configuration
config setup
        #plutodebug="control parsing"
        #plutodebug="all"
        dumpdir=/var/run/pluto/
        nat_traversal=yes
        virtual_private=%v4:10.0.0.0/8,%v4:192.168.0.0/16,%v4:172.16.0.0/12,%v4:25.0.0.0/8,%v6:fd00::/8,%v6:fe80::/10
        oe=off
        protostack=auto

conn l2tp-X.509
        authby=rsasig
        pfs=no
        auto=add
        rekey=no
        dpddelay=10
        dpdtimeout=90
        dpdaction=clear
        ikelifetime=8h
        keylife=1h
        type=transport
        # This is your server's IP:
        left=192.168.168.24  
        leftid=%fromcert
        leftrsasigkey=%cert
        # You will create this certificate:
        leftcert=/etc/ipsec.d/certs/ipsec-serverCert.pem
        leftprotoport=17/1701
        right=%any
        rightca=%same
        rightrsasigkey=%cert
        rightprotoport=17/%any
        rightsubnet=vhost:%priv,%no

conn passthrough-for-non-l2tp
        type=passthrough
        # This is your server's IP:
        left=192.168.168.24
        This is your server's gateway:
        leftnexthop=192.168.168.1
        right=0.0.0.0
        rightsubnet=0.0.0.0/0
        auto=route

Make sure that nat_traversal is on.

You need to provide the IP address of the interface that the IPsec connections will arrive on. If your IPsec server is behind a NAT (as in the diagram and example config), then this will be your private network IP and not the public IP.

The leftnexthop is the nexthop address for your openswan server. Typically you can determine that as such: 

# route -n 

Destination   Gateway        Genmask        Flags Metric Ref Use Iface
0.0.0.0       192.168.168.1  0.0.0.0        UG    0      0   0   eth0
192.168.168.0 0.0.0.0        255.255.255.0  U     0      0   0   eth0

In the above example, the gateway for the default network 0.0.0.0 is 192.168.168.1, which is the nexthop IP we are after.

If you put your server behind a NAT then you need to open UDP ports 500 and 4500 on your modem and pinhole those ports directly to the internal Openswan server IP (that you specified as "left"). This is a function frequently supported on even the most basic home routers.

Openswan - Special Notes 

type=transport

In IPsec configurations, normally you want "tunnel" over transport, but this is one of the rare times that "transport" is necessary.

rightca=%same

That's the same CA as the left (server) side. You specify this to indicate that only certificates signed by your IPsec CA are valid. You don't want anyone with a signed certificate from anywhere to get access!

L2TP / PPP

HowTo documents often recommend setting up /etc/xl2tpd/l2tp-secrets but this did not seem to work for me. In the end I needed to configure the /etc/ppp/chap-secrets file.

# cat /etc/ppp/chap-secrets
# Secrets for authentication using CHAP
# client   server                   secret   IP addresses
# *        *                        password 192.168.168.30
ipsec-user ipsec-server.example.com password 192.168.168.0/24

The beauty of this is that all four settings are needed, nothing is wide open. The username "ipsec-user" and the password "password" are filled in on the Windows client in the initial connection dialogue box. The "IP addresses" is the server local range.

# cat /etc/xl2tpd/xl2tpd.conf
; Output trimmed, generally
[global]                 ; Global parameters:
 port = 1701             ; * Bind to port 1701
 access control = no     ; * Refuse conn without IP match
[lns default]            ; Our fallthrough LNS definition
 exclusive = yes         ; * Permit one tunnel per host
 ip range = 192.168.168.100-192.168.168.105 ; Allocate
 hidden bit = no             ; * Use hidden AVP's?
 local ip = 192.168.168.24   ; * Our local IP to use
 length bit = yes            ; * Use length bit in payload?
 require chap = yes          ; * Require CHAP auth. by peer
 refuse pap = yes            ; * Refuse PAP authentication
 name = ipsec-server.example.com  ; * Use as hostname
 ppp debug = no                   ; * Turn on PPP debugging
 pppoptfile = /etc/ppp/options.l2tpd.lns  ; * ppp options

You need to take note of the "ip range", "local ip" and "name". In this example, the "ip range" comes from the same local IP network as the IPsec listening interface is on. This was selected as a matter of routing convenience.

# cat /etc/ppp/options.l2tpd.lns

lock
noauth
#debug
dump
logfd 2
logfile /var/log/xl2tpd.log
mtu 1400
mru 1400
ms-dns 192.168.168.1
lcp-echo-failure 12
lcp-echo-interval 5
require-mschap-v2
nomppe

If you have a local router then ms-dns is going to your local router if not then try whatever DNS address you normally use in your network.

Openssl - Config  

# ls -la /usr/lib/ssl/

lrwxrwxrwx  1 root root    20 Feb  1 23:16 openssl.cnf -> /etc/ssl/openssl.cnf

       countryName_default             = US
       stateOrProvinceName_default     = Oklahoma
       0.organizationName_default      = My Industries

Openssl - Server CA

Create a 10 year certificate authority (CA) specifically for IPsec.

# cd /etc/ipsec.d
# openssl req -x509 -newkey rsa:2048 -keyout private/caKey-ipsec-server.pem -out cacerts/caCert-ipsec-server.pem -days 3650

Openssl - Server Certificate

Now that we have a CA, we need to generate server certificates and have them signed by that CA. This sounds like two steps when one would do, but you may need to revoke your server certificate one day.

# openssl ca -in /etc/ipsec.d/private/ipsec-serverReq.pem -days 3650 -out /etc/ipsec.d/private/ipsec-serverCert.pem -notext -cert /etc/ipsec.d/cacerts/caCert-ipsec-server.pem -keyfile /etc/ipsec.d/private/caKey-ipsec-server.pem
# openssl rsa -in /etc/ipsec.d/private/ipsec-serverKey.pem -out ipsec-serverKey-rsa.pem

Openssl - Server CRL

Generate a Certificate Revocation List (CRL), which at this stage will have no revoked certificates. Later on, you will see how to easily revoke a signed certificate and add that certificate to this CRL.

# openssl ca -gencrl -keyfile /etc/ipsec.d/private/caKey-ipsec-server.pem -cert /etc/ipsec.d/cacerts/caCert-ipsec-server.pem -out /etc/ipsec.d/crls/ipsec-server.crl

Openssl - SSL Client Certificates 

The easiest method (in my opinion) is to issue all certificates from the Openswan server itself. I created a script (ipsec-cert-management.sh) to step through the certificate generation steps because it is unnecessarily painful.

When you run this script (ipsec-cert-management.sh), you'll be prompted for a series of passwords. One password will be for the CA's private key password, the other will be for the new private client key you are creating.

You should create a password for your client key. When you import the key into your Windows OS, you have to enter the password once, but other people will not be able to export and use the key elsewhere unless they know the password. In other words, there is little overhead in using a passworded key, but putting a password on the key adds an extra layer of difficulty for anyone trying to steal your private key from the local system.

Below, watch for my comments:

# New PK password
# CA password
# Enter

• Where "New PK password" is for the new client key that you are generating.
• Where "CA password" is the password for the CA private key"
• Where "Enter" means do nothing, just hit the enter key.

Here we go:

# ./ipsec-cert-management.sh testguy
Generating a 2048 bit RSA private key
..................................+++
............................+++
writing new private key to '/etc/ipsec.d/clientcerts/testguy/testguyKey.pem'
Enter PEM pass phrase:               # New PK password
Verifying - Enter PEM pass phrase:   # New PK password
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [US]:
State or Province Name (full name) [Oklahoma]:
Locality Name (eg, city) []:
Organization Name (eg, company) [My Industries]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:testguy  # Recommend using the client hostname
Email Address []:
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:       # Enter
An optional company name []:   # Enter
Using configuration from /usr/lib/ssl/openssl.cnf
Enter pass phrase for /etc/ipsec.d/private/caKey-ipsec-server.pem:   # CA password
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 13 (0xd)
Validity
Not Before: Mar 23 20:43:19 2014 GMT
Not After : Mar 20 20:43:19 2024 GMT
Subject:
countryName   = US
stateOrProvinceName   = Oklahoma
organizationName  = My Industries
commonName= testguy
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Comment:
OpenSSL Generated Certificate
X509v3 Subject Key Identifier:
X509v3 Authority Key Identifier:
keyid:
Certificate is to be certified until Mar 20 20:43:19 2024 GMT (3650 days)
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
Enter pass phrase for /etc/ipsec.d/clientcerts/testguy/testguyKey.pem:# New PK password
Enter Export Password:               # New PK password
Verifying - Enter Export Password:   # New PK password
Import /etc/ipsec.d/clientcerts/testguy/testguyKey.p12 to your Windows client.
total 24
drwxr-xr-x 2 root root 4096 Mar 23 21:42 .
drwxr-xr-x 3 root root 4096 Mar 23 21:42 ..
-rw-r--r-- 1 root root 1322 Mar 23 21:43 testguyCert.pem
-rw-r--r-- 1 root root 3526 Mar 23 21:43 testguyKey.p12
-rw-r--r-- 1 root root 1834 Mar 23 21:43 testguyKey.pem
-rw-r--r-- 1 root root  968 Mar 23 21:43 testguyReq.pem

I had written most of the content in the post before Heartbleed broke in April 2014. Until that point, I hadn't worried about digging into certificate revocation. In all reality, as a home user with an IPsec server running intermittently, I could count myself as particularly unlucky if someone had not only targeted my server for a Heartbleed attack but also managed to extract private keys or passwords; Noting also that my machine was patched within 24 hours of the announcement of Heartbleed.

Still, there will always be a need to be able to quickly revoke a certificate, for example if I thought that my windows machine had been compromised.

In that context, the script  (see end of post) needed to be enhanced to automate the revocation of any certificate signed by the IPsec CA. Note that the script cannot revoke a certificate signed by an external CA. In other words, if you generated a signed cert with this script then you can revoke the cert with this script.

Also note that if you sign a client certificate and subsequently generate a new CA certificate, you can still revoke the previously signed client certificate using the new CA certificate.

After revoking the client certificate and restarting IPsec, new connection attempts with the revoked cert will have this signature in the "ipsec barf" logs:

May  3 21:43:43 ipsec-server pluto[6922]: "l2tp-X.509"[1] 93.104.122.73 #1: Main mode peer ID is ID_USR_ASN1_DN: 'C=US, ST=Oklahoma, O=My Industries, CN=roadwarrior'
May  3 21:43:43 ipsec-server pluto[6922]: "l2tp-X.509"[1] 93.104.122.73 #1: certificate was revoked on May 03 19:42:27 UTC 2014
May  3 21:43:43 ipsec-server pluto[6922]: "l2tp-X.509"[1] 93.104.122.73 #1: X.509 certificate rejected
May  3 21:43:43 ipsec-server pluto[6922]: "l2tp-X.509"[1] 93.104.122.73 #1: no suitable connection for peer 'C=US, ST=Oklahoma, O=My Industries, CN=roadwarrior'
May  3 21:43:43 ipsec-server pluto[6922]: "l2tp-X.509"[1] 93.104.122.73 #1: sending encrypted notification INVALID_ID_INFORMATION to 93.x.x.x:61421
May  3 21:43:48 ipsec-server pluto[6922]: "l2tp-X.509"[1] 93.104.122.73 #1: Main mode peer ID is ID_USR_ASN1_DN: 'C=US, ST=Oklahoma, O=My Industries, CN=roadwarrior'

You can't revoke a certificate unless you have it to hand. I'd deleted a lot of certs during my testing and discovered that the index or "database" of issued certificates in /etc/ssl/demoCA/index.txt was populated with many certs that I no longer had. There appears to be no way to revoke a signed certificate by the serial number alone.

However, as we use the default openssl settings in the script  (see end of post) much of the time, the signed certificates are copied into /etc/ssl/demoCA/newcerts/ and named by serial number. As a result, we can revoke the serial number by finding the cert in that directory.

# ls -la /etc/ssl/demoCA/newcerts/
total 64
drwxr-xr-x 2 root root 4096 May  3 22:27 .
drwxr-xr-x 4 root root 4096 May  3 22:27 ..
-rw-r--r-- 1 root root 1744 Mar 17 20:32 01.pem
-rw-r--r-- 1 root root 1724 Mar 17 20:38 02.pem
-rw-r--r-- 1 root root 1724 Mar 17 20:49 03.pem
-rw-r--r-- 1 root root 1342 Mar 17 22:28 04.pem
-rw-r--r-- 1 root root 1342 Mar 17 22:29 05.pem
-rw-r--r-- 1 root root 1322 Mar 17 22:39 06.pem
-rw-r--r-- 1 root root 1342 Mar 17 23:38 07.pem
-rw-r--r-- 1 root root 1322 Mar 18 21:06 08.pem
-rw-r--r-- 1 root root 1367 Mar 18 23:09 09.pem
-rw-r--r-- 1 root root 1330 Mar 23 14:21 0A.pem
-rw-r--r-- 1 root root 1322 Mar 23 21:40 0B.pem
-rw-r--r-- 1 root root 1322 Mar 23 21:42 0C.pem-revoked
-rw-r--r-- 1 root root 1322 Mar 23 21:43 0D.pem
-rw-r--r-- 1 root root 1330 May  3 22:27 0E.pem

# cat /etc/ssl/demoCA/index.txt
V   240314193233Z   01  unknown /C=US/ST=Oklahoma/O=My Industries/CN=ipsec-server.example.com
V   240314193805Z   02  unknown /C=US/ST=Oklahoma/O=My Industries/CN=toshiba
V   240314194925Z   03  unknown /C=US/ST=Oklahoma/O=My Industries/CN=toshiba
V   240314212827Z   04  unknown /C=US/ST=Oklahoma/O=My Industries/CN=ipsec-server.example.com
V   240314212952Z   05  unknown /C=US/ST=Oklahoma/O=My Industries/CN=ipsec-server.example.com
R   240314213921Z   140503193859Z   06  unknown /C=US/ST=Oklahoma/O=My Industries/CN=toshiba
R   240314223758Z   140503194035Z   07  unknown /C=US/ST=Oklahoma/O=My Industries/CN=ipsec-server.example.com
V   240315200605Z   08  unknown /C=US/ST=Oklahoma/O=My Industries/CN=hnzlwin7
R   240320132104Z   140503194227Z   0A  unknown /C=US/ST=Oklahoma/O=My Industries/CN=roadwarrior
V   240320204055Z   0B  unknown /C=US/ST=Oklahoma/O=My Industries/CN=testguy
R   240320204223Z   140503200045Z   0C  unknown /C=US/ST=Oklahoma/O=My Industries/CN=testguy
R   240320204319Z   140503152636Z   0D  unknown /C=US/ST=Oklahoma/O=My Industries/CN=testguy
V   240430202749Z   0E  unknown /C=US/ST=Oklahoma/O=My Industries/CN=roadwarrior

To verify the loading of the CRL, "ipsec barf" will give you something like this:

000 May 03 21:43:35 2014, revoked certs: 4
000issuer:  'C=US, ST=Oklahoma, O=My Industries, CN=ipsec-server.example.com'
000distPts: 'file:///etc/ipsec.d/crls/ipsec-server.crl'
000updates:  this May 03 21:42:30 2014
000  next Jun 02 21:42:30 2014 ok

and this...

May  3 21:43:35 ipsec-server pluto[6922]: Changing to directory '/etc/ipsec.d/crls'
May  3 21:43:35 ipsec-server pluto[6922]:   loaded crl file 'ipsec-server.crl' (759 bytes)
May  3 21:43:35 ipsec-server pluto[6922]: loading certificate from /etc/ipsec.d/certs/ipsec-serverCert.pem
May  3 21:43:35 ipsec-server pluto[6922]:   loaded host cert file '/etc/ipsec.d/certs/ipsec-serverCert.pem' (1342 bytes)

The output of the ipsec-cert-management.sh script (when revoking) will look something like this:

# ./ipsec-cert-management.sh /etc/ssl/demoCA/newcerts/02.pem
/etc/ssl/demoCA/newcerts/02.pem exists. Do you want to revoke this Certificate? [y/N]: y
Using configuration from /usr/lib/ssl/openssl.cnf
Enter pass phrase for /etc/ipsec.d/private/caKey-ipsec-server.pem:
Revoking Certificate 02.
Data Base Updated
Using configuration from /usr/lib/ssl/openssl.cnf
Enter pass phrase for /etc/ipsec.d/private/caKey-ipsec-server.pem:
Certificate Revocation List (CRL):
Version 2 (0x1)
Signature Algorithm: sha1WithRSAEncryption
Issuer: /C=US/ST=Oklahoma/O=My Industries/CN=ipsec-server.example.com
Last Update: May  3 22:21:19 2014 GMT
Next Update: Jun  2 22:21:19 2014 GMT
CRL extensions:
X509v3 CRL Number:
13
Revoked Certificates:
Serial Number: 02
Revocation Date: May  3 22:21:15 2014 GMT
Serial Number: 06
Revocation Date: May  3 19:38:59 2014 GMT
Serial Number: 07
Revocation Date: May  3 19:40:35 2014 GMT
Serial Number: 09
Revocation Date: May  3 19:46:03 2014 GMT
Serial Number: 0A
Revocation Date: May  3 19:42:27 2014 GMT
Serial Number: 0C
Revocation Date: May  3 20:00:45 2014 GMT
Serial Number: 0D
Revocation Date: May  3 15:26:36 2014 GMT
Serial Number: 0F
Revocation Date: May  3 22:00:11 2014 GMT
Serial Number: 10
Revocation Date: May  3 22:02:53 2014 GMT
Signature Algorithm: sha1WithRSAEncryption
Done, check for errors above. YOU MUST RESTART IPSEC!
-rw-r--r-- 1 root root 1724 Mar 17 20:38 /etc/ssl/demoCA/newcerts/02.pem-revoked
My advice is to make sure you have (R)evoked anything you aren't sure about. At the bare minumum you will have one (V)alid certificate for your server and one (V)alid certificate for an IPsec client.
# cat /etc/ssl/demoCA/index.txt
R   240314193233Z   140503223446Z   01  unknown /C=US/ST=Oklahoma/O=My Industries/CN=ipsec-server.example.com
R   240314193805Z   140503222115Z   02  unknown /C=US/ST=Oklahoma/O=My Industries/CN=toshiba
R   240314194925Z   140503223056Z   03  unknown /C=US/ST=Oklahoma/O=My Industries/CN=toshiba
R   240314212827Z   140503223459Z   04  unknown /C=US/ST=Oklahoma/O=My Industries/CN=ipsec-server.example.com
V   240314212952Z   05  unknown /C=US/ST=Oklahoma/O=My Industries/CN=ipsec-server.example.com
R   240314213921Z   140503193859Z   06  unknown /C=US/ST=Oklahoma/O=My Industries/CN=toshiba
R   240314223758Z   140503194035Z   07  unknown /C=US/ST=Oklahoma/O=My Industries/CN=ipsec-server.example.com
R   240315200605Z   140503223122Z   08  unknown /C=US/ST=Oklahoma/O=My Industries/CN=hnzlwin7
R   240315220911Z   140503194603Z   09  unknown /C=US/ST=Oklahoma/O=My Industries/CN=hnzlwin7/emailAddress=hnzlwin7@hnzl.de
R   240320132104Z   140503194227Z   0A  unknown /C=US/ST=Oklahoma/O=My Industries/CN=roadwarrior
R   240320204055Z   140503223138Z   0B  unknown /C=US/ST=Oklahoma/O=My Industries/CN=testguy
R   240320204223Z   140503200045Z   0C  unknown /C=US/ST=Oklahoma/O=My Industries/CN=testguy
R   240320204319Z   140503152636Z   0D  unknown /C=US/ST=Oklahoma/O=My Industries/CN=testguy
V   240430202749Z   0E  unknown /C=US/ST=Oklahoma/O=My Industries/CN=roadwarrior
R   240430215836Z   140503220011Z   0F  unknown /C=US/ST=Oklahoma/O=My Industries/CN=nogood
R   240430220142Z   140503220253Z   10  unknown /C=US/ST=Oklahoma/O=My Industries/CN=nogoodagain

With this ipsec-cert-management.sh script, it's easy to add and revoke a certificate, so there's no excuse to compromise security on the grounds of complexity.

References

1. Using a Linux L2TP/IPsec VPN server with Windows Vista 
2. Github: https://github.com/mofftech/Server-Automation/

HowTo: Configure Exim to scan zip attachments for malicious files

This post provides a quick way of configuring zip attachment scanning to Exim. I've drawn this example from a number of questions across the web and hopefully enhanced the experience by sharpening up the scripting and explaining in more detail what is happening. 

I'd intended today to cleave this information into the Making a Mailserver (Part 5) - Wonderful Spam part of my series on setting up a personal mailserver. However it seemed better to split this howto out as a separate entity to avoid confusion and lessen the difficulty of an already complex series.

We will only scan the contents of the zip file and look at file extensions. We won't extract files and scan the files in any way. Extracting and scanning for filetypes or passing through an anti-virus application is a better way of identifying malicious files. Scanning files is a CPU intensive activity and fraught with new complications such as whether your anti-virus application just introduced a new attack vector into your environment, I'm looking at you, Symantec.

The reality is that mass email Malware (not Spear Phishing for example) relies on the fact that people blindly double-click on the file they received, rather than following complicated manoeuvres to rename or open a file according to the attacker's in-message instructions. In that light, simply checking the extensions within a zip file is good enough for me.

Dropping emails with files that are not zipped

This is easily customised with Exim's ACLs.

conf.d/acl/40_exim4-config_check_data

deny message = Please don't email me attachments 
 demime = bat:btm:cmd:com:cpl:dll:exe:lnk:msi:pif:prf:reg:scr:vbs:url:wsf:docm:hta:jse

Mind out for the line wrapping above, that should just be two lines with "deny message ..." and "demime ..." as the start of each line. It should be obvious to you how to add or remove extensions from demime.

This is a hard drop. The sender is told to go away, rather than to try again later.

Each demime entry is the file extension of a file attachment. Notice that I have not put zip, tar and tgz in there. Zip files we check next.

Zip files are still a normal part of day to day email, we can't just drop them. Dmarc xml for example is sent as a zip attachment.

Dropping emails with zipfiles that contain malicious attachments

Again, we are in the check_data ACL.

conf.d/acl/40_exim4-config_check_data

deny message = Please don't email me attachments
     log_message = DENY: zip with blocked content
     demime      = zip
     condition   = ${run{/etc/exim4/scan_zip.sh $message_id}{0}{1}}

Mind out for line wrapping, this is a four line stanza, the last line is the "condition" line.

Again, this is a hard drop. The sender is told to go away, rather than to try again later. The log_message is what you will see in the /var/log/exim4/rejectlog. "demime" looks for an attachment with the ".zip" extension and triggers the rule which fires the "condition".

The "condition" runs a shell script that must return either 0 or 1. "0" means no problem while "1" triggers the "deny".

The scan_zip.sh script itself you can download from my git location here. Verify that logger, grep, unzip and ls are installed and that the path to them has been correctly set in the script according to your system. You can easily add or remove file extensions in that script.

You might notice that I drop when a zip file is seen in a zip file. That's an overly cautious approach, but I see no genuine reason for that to occur. It certainly could occur, if someone zips you up a pile of files that also includes a zip archive. You might want to remove that from the script or even extract and scan the zip in the zip, it's easily done and it's your call.

Verifying

I recommend initially to tail the /var/log/exim4/rejectlog file and watch for messages being dropped. Look at the "rejected after DATA" information, this is where the log_message output is appended.

Check daily the messages that are being dropped, I recommend a grep such as:

# grep -E 'DATA|X-Spam-Score|Subject:|To:' /var/log/exim4/rejectlog

If something goes wrong in your shell script, it will exit with a non zero value and trigger the Exim ACL deny. Be careful to check that the script runs cleanly.

Temporarily uncomment the "#$cmd_logger" messages and watch your syslog (normally /var/log/syslog) for the pass and fail messages. The logger messages give you more detail on what and why something was denied. If you have systemd, then the "service exim4 status" command will give you a recent history of messages that Exim wrote to syslog.

Postscript

To date I've never received Malware delivered in ".tar" or ".tgz" files. When that day comes, I am going to write new deny rules to inspect the contents just as I do with ".zip". I'll probably blog about that here!

Making a Mailserver (Part 6) - Server Migration & Backup

This is an instalment in my series on setting up a Linux based mailserver. See these posts:

• Making a Mailserver (Part 1) - What's in a Name
• Making a Mailserver (Part 2) - My Personal Daemons: Exim and Dovecot
• Making a Mailserver (Part 3) - Let's Encrypt Webmail
• Making a Mailserver (Part 4) - Credibility is Critical
• Making a Mailserver (Part 5) - Wonderful Spam
• Making a Mailserver (Part 6) - Server Migration & Backup

In this post, I cover the steps you must plan for when migrating your mail services from one server to another. Plan now for a migration before everything is a distant memory. I won't get into deep technical detail here because the technical detail is covered in the previous posts.

Careful Planning Eases the Transition

While first setting up your mailserver, make copious notes for speedier disaster recovery. After setting up a server of any kind, I like to dump the history command to file and save that off-box.

Something to ponder: could you recreate your server from the backups alone?

Backups

If your VPS provider isn't taking care of your backups (this often comes at an additional cost) then you really should backup frequently. If you're a home-brew error-prone control-freak like me, you'll want to custom script the backup process because the job is otherwise tedious and you'll never get around to it.

If you've followed my posts, then you need to backup the following directories completely:

• /etc/
• /home/
• /var/www/
• /usr/lib/cgi-bin/
• /var/mail/

The active inbox files are in /var/mail while the user's mail folders will be stored in their /home path. Backing up those can take some time.

Leave no configuration file behind and grab /etc, this directory doesn't take a lot of space.  The /var/www and /usr/lib/cgi-bin/ dirs are just in case you were messing around with custom web pages despite my constant pleas that you don't.

You might also want to collect a list of all installed packages.

Over time I developed a script to take care of this and scp the backups to a remote server. You can easily adapt it for your needs. Grab my backup-server.sh script from github.com if you like.

Warning! If you do use my backup-server.sh script, be aware that using ssh-copy-id is unwise unless you secure your private key with a password when creating it (ie. with ssh-genkey). The reason is that if you are using a cloud based VPS, the Administrators at that VPS provider can clone your server and access any file they please. If your VPS is installed on an encrypted volume, that's great, but a hacker who compromises your running VPS will also have access to your keyfiles.

Hence I strongly recommend you do one of two things:

1. Put a password on your private ssh key when generating it.
2. Run the script using a one-off key:
• Create a passwordless one-off key to copy to the remote server.
• Run the script.
• Delete the local keys (in ~/.ssh/) and the public key from the remote server (in ~/.ssh/authorized_keys).

I've been considering scripting option 2.

Why Would I Need to Migrate?

In my experience, migrations are done only because your hand has been forced and the decision is out of your control. To name a few obvious reasons:

1. You lost your nerve and decided to migrate the data from the clapped-out 486 in your shed to a VPS in the cloud.
2. You found a better service for half the price.
3. Your existing VPS provider declared that their system upgrade requires you to migrate to a new VPS. One VPS provider pulled this trick on me twice in two years and it's very annoying.
4. Your Operating System went End of Software Support. Running a distribution upgrade is nerve-wracking to the point that I prefer to rebuild from scratch from a clean install.
5. You prefer to operate as the root user and ran "rm -rf /".
6. Your system has been compromised.

Migration Steps

Server migration involves, these steps in this order:

1. Prepare the new server by installing all the packages required by your old server. 
2. Configure the new server. You'll be recycling the config files from the old server to do that. You can recycle the FQDN and keep the same hostname, exercising some caution to not get the two confused.
3. Make sure that the users from your old server /etc/passwd & /etc/group are inserted into the new server using the same uid and gid otherwise /home and /var/mail/ is going to be a mess. Don't forget to check /etc/aliases.
4. Ensure the new server doesn't have the mail daemon (postfix/sendmail/exim) started. 
5. Backup the old server.
6. Take the old server mail daemon offline (turn off postfix/sendmail/exim).
7. Update the MX record to point to the new server, but keep the new server's mail daemon offline.
8. Copy all the mailboxes and user directories over to the new server. It's easier to tar and compress these (preserving file ownerships and permissions) and then copy, than to scp the files in place. You need to copy many things as root while still preserving the file ownerships and permissions.
9. Start the mail daemon on the new server. 
10. Fix all the problems.  
11. Once you're really sure that the new server is working as expected, shut down the old one. Goto 10.

Here's why you want to do those steps in order:

• You don't want to have two mailservers running at the same time.
• The DNS changes take hours to propagate, so as long your old mailserver is offline, you might as well do the DNS change straight away.
• You make the copy of mailboxes and user directories with the knowledge that they cannot change and the version you are copying is the most current.

It's okay for your mail domain to go dark for a few hours. The Internet doesn't really mind. Most mail hosts make a few attempts over several hours before giving you up for dead. Interestingly, I've noticed that Google / Gmail is very quick to notice your MX server change (less than 30 minutes).

Data Security in the Cloud

As I mentioned earlier, unless you are running from an encrypted volume, your data is open to the VPS company's admins. Either you encrypt your volume or you have no offline data security. You need to think about what will happen to your data when you retire your old server instance.

I know of no elegant solution to the problem of protecting the data on an abandoned VPS. The data security policy of a VPS company should inform your choice of provider.

If you're on a cloud VPS, delete all of the sensitive directories on the old server; likely the same directories that you would backup. It seems possible to me that your VPS provider could still recover and steal the data. There's also a chance that once the VPS provider assigns the disk area to another customer then that customer may be able to recover files.

You could delete the sensitive data and then fill the disk to capacity with junk. Redirecting the output of "yes" to file would be a quick way to do it. An ultra paranoid approach would be do that several times.

Making a Mailserver (Part 5) - Wonderful Spam

This is an instalment in my series on setting up a Linux based mailserver. See these posts:

• Making a Mailserver (Part 1) - What's in a Name
• Making a Mailserver (Part 2) - My Personal Daemons: Exim and Dovecot
• Making a Mailserver (Part 3) - Let's Encrypt Webmail
• Making a Mailserver (Part 4) - Credibility is Critical
• Making a Mailserver (Part 5) - Wonderful Spam
• Making a Mailserver (Part 6) - Server Migration & Backup

In this post we setup Spamassassin with exim4 to stomp on as much spam as possible. We won't give anything that looks like spam a chance to be delivered, we'll dump such messages before they even complete the delivery process.

And yet it spams

Why Spamassassin?

There are alternatives, a good friend of mine recently recommended dspam to me, so that's on my list to investigate. Spamassassin doesn't catch everything, mostly because some spammers are pretty damn good at their job. It does not do a great job of spotting messages delivering Malware and the tediously regular emails from bearing manufacturers.

If there was such a thing as a perfect solution, it wouldn't be by implementing just one technology. Nothing's perfect. I chose Spamassassin because it is mature, well understood, backed by Apache and easy to setup.

This is Kinda Interesting

For a good number of years I went without receiving a lot of spam. I didn't intentionally publish my email address on the web and many websites take care to obscure your email address, for example, when publishing mailing list archives.

I was intrigued by Steve Gibson's assertion in Security Now #557 that it takes multiple years before your email address really gets on Spammers' radar. He changes his email address once a year:

And something as simple as changing your email address loses spam. That is, it's just gone. And you might think that, oh, it's going to find you again within a week or two. No. It takes, I can attest to this, years, multiple years.

I'd naively held the belief that Spammers found your domain name and then worked through a list of common names to mail to (michael@, john@, chris@, etc). Perhaps they do do that, but it seems that scraping websites and database dumps is the most common and less time-wasting way of building a list of recipients.

In 2015 the level of spam quickly started to get out of control for me. Malware especially was really flooding in and even though I am largely Linux and Android focussed, constantly deleting spam made checking my email a tedious, rather than fun, task.

Plus one for Thunderbird however. I took the time to train Thunderbird's junk mail handling and it is really good. However, as I check email on my phone (most of the time), Thunderbird's junk handling wasn't going to help unless I always had it running in the background on some workstation, somewhere.

The spam was pouring in and since I was redirecting (aliasing via /etc/aliases) some mailboxes to Gmail accounts, I was ending up with a large queue of frozen messages because Gmail was not happy to handle redirected Spam. If you want a Spam free mailbox, there's arguably nothing better than Gmail. I was worried about the potential damage I was doing to my mail server's reputation with Gmail by reflecting Spam straight to Gmail.

Enter Spamassassin.

Exim and Spamassassin Integration

I recommend you first review the debian exim wiki. I did find it to be wrong in places when I referred to it. There was some Exim documentation about acl actions that also informed my config. I'll be providing a multi mail-domain example because I do handle mail for multiple domains. The following example will work for one mail domain or many.

I use exim4 split config files because ... that's what everyone else does.

In brief, the debian exim wiki tells you to do the following:

#apt-get install spamassassin

If you are using Debian Jessie or later (with systemd enabled by default), enable and start the service using systemctl;

#systemctl enable spamassassin.service

On earlier Debian releases, edit /etc/default/spamassassin ...

ENABLED=1 

...and then start the daemon.

#/etc/init.d/spamassassin start

At this point I found divergences between what the documentation tells you to do and what works in reality. The "add_header" did not work for me, following the wiki instructions. Here's how I set it up:

/etc/exim4/conf.d/acl/40_exim4-config_check_data

# warn
#   spam = Debian-exim:true
#   message = X-Spam_score: $spam_score\n\
#             X-Spam_score_int: $spam_score_int\n\
#             X-Spam_bar: $spam_bar\n\
#             X-Spam_report: $spam_report
#
# put headers in all messages (no matter if spam or not)
  warn  spam = Debian-exim:true
      add_header = X-Spam-Score: $spam_score ($spam_bar)
# add second subject line with *SPAM* marker when message
# is over threshold
  drop  spam = Debian-exim
#      add_header = Subject: ***SPAM (score:$spam_score)*** $h_Subject:

Important Points:

• The debian docs "Subject" manipulation simply did not work for me. Refer to the "Rewriting Subject" section further below.
• The debian docs used "nobody" as the user, I changed this to Debian-exim. Using "nobody" gets you all kinds of painful log messages.
• The debian docs used "add_header = X-Spam-Report: $spam_report" on all  messages, this resulted in a message in the header of every email saying that the email had been detected as spam, regardless of the score. 
• I do still insert the X-Spam_Score in every message.
• I'm dropping anything over the Spamassassin threshold (required_score). The incoming message will be "rejected after DATA".

You will want to tinker with the threshold on dropped messages. 8 is too high, but it's better to start high and then inspect the score on the spam that makes it through. The bulk of spam appears to get very high scores, but between 4 to 5 there is a crossover between legitimate email and spam.

You also receive spam that gets scores as low as 1 and it's impossible to filter at that level without losing a lot of legitimate email.

You can tinker with the required score in /etc/spamassassin/local.cf and the default at the time of writing is 5, which I think is about right.

required_score 8.0

It's important to remember that the delivery agent is going to get a hard fail when a message scores over the required_score. It probably won't come back for a second try. A slighted mailing list server, for example, may mark your address as a hard fail and remove your subscription.

The rewrite_header in the Spamassassin config is meaningless because Exim is handling the mail and just asking Spamassassin for its opinion on the spam score. Other elements in the Spamassassin file are relevant to scoring the message.

That's it! That's all you need to do.

Rewriting Subject

I didn't implement this because I elected to dump the high scoring messages and write (for every message) the X-Spam-Score to the headers.

Reviewing the Efficacy

You really must spend days or weeks checking in with your Exim logs in addition to reviewing the Spam messages that are slip through.

• When looking at the spam that hits your inbox, take a look at the X-Spam-Score header that was written in by Spamassassin. View the message source to see the headers. 

• Don't be confused by  fake headers added by the spammer, such as fake Spam Score information.

• There is often false information in the headers about being checked by this or that antivirus software. 

• Message headers should be read from bottom to top. Each mail agent prepends its headers to the top of the message as the message bounces around mailserver to mailserver.

Review the Exim reject log. Here's what you should see when things are working:

# tail -vf /var/log/exim4/rejectlog 

2016-07-06 18:41:41 1bKptM-0006PS-7x H=208-180-142-165.chstcmtk01.com.sta.suddenlink.net [208.180.142.165] F=<xxx@swisslens.com> rejected after DATA
Envelope-from: <xxx@swisslens.com>
Envelope-to: <xxx@moff.tech>
P Received: from 208-180-142-165.chstcmtk01.com.sta.suddenlink.net ([208.180.142.165])
        by moff.tech with smtp (Exim 4.84_2)
        (envelope-from <xxx@swisslens.com>)
        id 1bKptM-0006PS-7x
        for xxx@moff.tech; Wed, 06 Jul 2016 18:41:40 +0200
  Date: Wed, 06 Jul 2016 14:35:35 -0300
F From: "CamilleHot" <xxx@mndistaog.org>
R Reply-To: "CamilleHot" <xxx@mndistaog.org>
  X-Priority: 3 (Normal)
I Message-ID: <75461.14671320@mndistaog.org>
T To: xxx@moff.tech
  Subject: Come here! I want to make love to you
  MIME-Version: 1.0
  Content-Type: multipart/alternative;
        boundary="605311998876970"
  X-Spam-Score: 18.3 (++++++++++++++++++)

Notice the X-Spam-Score of 18.3 - high but not off the charts. Let's review the kind of scores we've recently seen:

# grep X-Spam-Score /var/log/exim4/rejectlog

  X-Spam-Score: 7.5 (+++++++)
  X-Spam-Score: 7.2 (+++++++)
  X-Spam-Score: 11.8 (+++++++++++)
  X-Spam-Score: 8.2 (++++++++)
  X-Spam-Score: 20.0 (++++++++++++++++++++)
  X-Spam-Score: 14.2 (++++++++++++++)
  X-Spam-Score: 18.4 (++++++++++++++++++)
  X-Spam-Score: 15.4 (+++++++++++++++)
  X-Spam-Score: 5.1 (+++++)
  X-Spam-Score: 16.8 (++++++++++++++++)
  X-Spam-Score: 13.6 (+++++++++++++)
  X-Spam-Score: 6.5 (++++++)
  X-Spam-Score: 14.2 (++++++++++++++)
  X-Spam-Score: 9.5 (+++++++++)
  X-Spam-Score: 18.5 (++++++++++++++++++)
  X-Spam-Score: 18.3 (++++++++++++++++++)
  X-Spam-Score: 6.1 (++++++)

In the period where you're still finding the right score level, I would recommend going back and looking at the headers on the 5.1 message: use "less" to simply view the file and search for the score string.

In fact I discovered, as I wrote this, that the 5.1 in the example above was a legitimate email about a delivery I was waiting on. Oh dear, perhaps I'll nudge the required_score up to 5.1. In my experience, 5.5 is too high.

It's easy to review the scores on the messages in your mailboxes. Scores can actually be negative in number, the lowest I've noticed is -11.89:

Your current inbox:

# grep X-Spam-Score /var/mail/you

Other folders:

# grep X-Spam-Score /home/you/mail/Trash

Keep looking at the Exim logs. Be curious. Learn what the headers mean. Tinker. It's fascinating stuff.

References

http://michaelfranzl.com/2015/02/11/exim-spamassassin-rewriting-subject-lines-adding-spam-score
https://spamassassin.apache.org/full/3.1.x/doc/Mail_SpamAssassin_Conf.html
http://www.exim.org/exim-html-current/doc/html/spec_html/ch-access_control_lists.html
http://www.exim.org/exim-html-current/doc/html/spec_html/ch-content_scanning_at_acl_time.html
https://www.maretmanu.org/homepage/inform/exim-spam.php