How to get the SSL/TLS certificate chain right

After installing Firefox and Chrome on a new PC, I noticed that I was getting "issuer unknown" (Firefox: SEC_ERROR_UNKNOWN_ISSUER) errors on a website that I was checking connectivity against. The website was one that I was recently put in charge of and the organisation had a paid-for COMODO certificate for a Wordpress install.

On other computers there were no such errors reported by Firefox or Chrome for this website, so I initially missed the significance of the problem.

Background

Note: "Certificate" should be read as "public certificate". The private cert or key is not discussed in this document.
 
Websites that support HTTPS require a valid SSL/TLS certificate or the client will receive certificate warnings from the application or browser. Happily, your community organisation or personal website can get by with a free certificate courtesy of Let's Encrypt. At the same time, the website needs to provide a certificate chain, which essentially informs the client (your browser) about the identity of the host that signed your certificate.

Likewise, the certificate of the host that signed the previous host's certificate needs to be provided. This recursive method of providing the certificate of the previous "intermediary" signer continues until the root certificate authority (CA) is reached. The root CA certificate does not need to be provided because it should be specifically trusted on the client software or browser or the whole infrastructure of trust is useless. Browsers and other software will ship with root CA certificates, or you can manually add them if necessary.

Any intermediary CA needs to be included in the certificate chain, but the root CA should not be included.

Incomplete, Contains anchor

I turned to Qualys SSL Labs to see whether I could obtain a head start on the problem. I saw warnings, which told me where to look but didn't help in identifying exactly what was wrong.

"This server's certificate chain is incomplete. Grade capped to B."

And later in the report:

"Chain issues - Incomplete, Contains anchor"

 I had somewhere to investigate at least - the certificate chain.

Also worth noting was that curl also complained of certificate problems on my desktop even if Firefox and Chrome did not.

Examining the Certificate Chain

Note: example.com is a placeholder for the real URL I was investigating.
 
openssl is the obvious tool to turn to for seeing the nitty-gritty of an SSL/TLS session. I was able to easily view the certificate chain:

$ openssl s_client -connect www.example.com:443
...
Certificate chain

 0 s:/OU=Domain Control Validated/OU=Hosted by webgo GmbH/OU=PositiveSSL/CN=www.example.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA

 1 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root

 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root

To the untrained eye, the above output looks incomprehensible. But with a little understanding and research the problem can be clearly seen. The chain consists of three certificates (0,1,2) issued by the website. Each certificate has a (s)erver that the certificate belongs to and an (i)ssuer that signed the certificate for that server.

The first certificate (0) is for www.example.com and it was issued by COMODO RSA Domain Validation Secure Server CA. 

The second certificate (1) is expected to be for the COMODO issuer in the first certificate, but it is not. The certificate is for some other server AddTrust External CA Root, who is oddly also the issuer (a so called "self signed" certificate). Things are broken to bits from this point.

The third certificate (2) is completely superfluous because the second cert in this chain should not be there. In this third cert, we see it is for the server COMODO RSA Certification Authority and has been signed by the issuer AddTrust External CA Root.

Fixing the Mess

The solution was to provide a correct certificate chain. The first certificate (our certificate) was valid, but since it was signed by "COMODO RSA Domain Validation Secure Server CA" that needs to be the next public certificate found in the chain.

I first checked whether this was part of a typical Firefox CA set. The shipped CA certs can be viewed either on the Mozilla website or via the options->preferences of Firefox itself:

Yeah that's in German, sorry, but the English version will look basically the same.

In this case, notice that in the above screenshot "COMODO RSA Domain Validation Secure Server CA"  is actually in my certificate list in Firefox. I realised afterwards that at some point I had clicked through the invalid certificate warnings and added the certificate to my Firefox certificate store, to be trusted for next time. That's why I only noticed the problem after installing a new PC with Firefox and Chrome.

Just to repeat myself, the "COMODO RSA Domain Validation Secure Server CA" certificate is not part of the default suite of certificates trusted by Firefox. I needed to download the public certificate from COMODO here and tell apache2 (the site's web server) to use that certificate and only that certificate as part of the certificate chain.

Briefly, this meant configuring these apache2 settings ...

SSLCertificateKeyFile /etc/apache2/ssl.key/www.example.com.key
SSLCertificateFile /etc/apache2/ssl.crt/www.example.com.crt
SSLCertificateChainFile /etc/apache2/ssl.ca/www.example.com.ca

... ensuring that SSLCertificateKeyFile  contained only the private key of the server, SSLCertificateFile contained only the public certificate associated with the aforementioned private key and that SSLCertificateChainFile contained only the public certificate for "COMODO RSA Domain Validation Secure Server CA". If you have multiple layers of signing, you need to add each intermediary CA to this file, in the correct order. 

This works because the certificate for the intermediary "COMODO RSA Domain Validation Secure Server CA" is issued by "COMODO RSA Certification Authority" and that root CA is part of the shipped set of certificates for Firefox, Chrome, curl, openssl and any other SSL/TLS client you care to name.

The certificate chain now looks like this, (see earlier openssl command syntax):

Certificate chain

 0 s:/OU=Domain Control Validated/OU=Hosted by webgo GmbH/OU=PositiveSSL/CN=www.example.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
 
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority

In the first certificate, the (i)ssuer name is the (s)erver name of the second certificate. The second validates the first. There is no need for a third certificate because the issuer of the second certificate is part of an established set of known root Certificate Authorities.

Switching Your Drupal to HTTPS

This post describes the steps I undertook to take to switch my Drupal site over to HTTPS. I'd made several attempts at this after getting a certificate via Let's Encrypt but ran into problems with mixed content, meaning that Firefox would not render the page properly while there was a mixture of encrypted and non-encrypted objects being loaded from the website.

I finally found the time to sit down and work it out. There is a nice explanation on the Drupal website and to a large extent I followed the 'best possible' solution there. My post adds some extra points you should know.

Let's Encrypt

I've covered the topic of Let's Encrypt in some detail in other places on this blog, so there's no need for me to cover this off. Get your certificate from Let's Encrypt and make sure that your webserver (in my case apache2) is using the certificate and responds to the https:// form of your Drupal website.

Go Full Encryption

As I mentioned above, the explanation on the Drupal website is a good place to start. I fiddled around with the Secure Login module and also with using $conf['https'] = TRUE; in the settings.php file.  In the end the "best possible security" option was the simplest and strongest solution.

Redirect Everything To HTTPS

Take the information from the Drupal website and implement the VirtualHost configuration to redirect all HTTP to HTTPS. Invoking the "Redirect" example on the VirtualHost, rather than the "Rewrite" example is easier and more elegant.

Mixed Content Problems

It's at this point that you may encounter mixed content problems, meaning that the Drupal site will not render correctly and have a yellow icon where the green padlock should be in the URL bar.

There are two things to do. The first thing is to check for objects that are actually loading over HTTP. If you're using Firefox, press F12 to load firebug and you can audit/search the source of the page for "http" objects. Do not confuse an http link to an external site (such as in an <a href..> tag) with content that is actually loading.

Mixed content means that your page has actually caused the browser to download an object from over HTTP instead of HTTPS. In my case I discovered a logo image present on every page had a hard-coded http:// address. Once I fixed that, I still had mixed content errors.

The next thing to change is the $base_url of your Drupal site, to indicate a default to https on all urls. This should have the effect of changing all the relative links that Drupal generates on the fly when rendering pages.

/etc/drupal/7/sites/moff.tech/settings.php:

$base_url = 'https://moff.tech';

In my case, these two steps got me the green padlock. It's possible that your install could have hard-coded HTTP object sources inside pages. All of that would need to be tidied up and depending on the size of your site a tedious exercise.

Good luck and please do share here any tips (war stories) from your experience!

Making a Mailserver (Part 3) - Let's Encrypt Webmail

This is an instalment in my series on setting up a Linux based mailserver. See these posts:

• Making a Mailserver (Part 1) - What's in a Name
• Making a Mailserver (Part 2) - My Personal Daemons: Exim and Dovecot
• Making a Mailserver (Part 3) - Let's Encrypt Webmail
• Making a Mailserver (Part 4) - Credibility is Critical
• Making a Mailserver (Part 5) - Wonderful Spam
• Making a Mailserver (Part 6) - Server Migration & Backup

You May Not Want Webmail, But...

Consider webmail as an optional section, Let's Encrypt as mandatory. Webmail is not something that I recommend you set up on your mailserver, but webmail is handy as a when all else fails backup. When you're travelling, you can't be sure that you can SSH to your mailserver. Protocols outside of HTTP, HTTPS and DNS are often blocked, which makes mutt not an option. When visiting friends and family, I've found mail clients can be forbidden SMTP or IMAP protocol access to locations outside of the ISP.

On the downside, setting up a web server uses valuable server resources, increases complexity and widens your security exposure.

On the security point, I use squirrelmail and honestly I don't know that it's a very good option. I think squirrelmail is easy to set up and lightweight to run, but development appears to have stopped, which means that security updates won't be arriving. On the other hand, from the remote attacker standpoint if an attacker doesn't already have the username and password of a mail user, then they only have the squirrelmail login page to hack away at. Well hackers are inventive, there should be other avenues into it than just that page. An attacker could craft a malicious message, targeted to exploit squirrelmail, that a user might open via squirrelmail. It's an acceptable risk to me, for now.

If you're not interested in setting up a webmail front-end, what this section will additionally do is provide an easier path to set up Apache for the generation of Let's Encrypt certificates to use with your IMAP and SMTP server daemons.

Apache

Apache will host the squirrelmail install or, at the very least, host the method by which you can easily generate a certificate for your mailserver via letsencrypt. If only required for the latter option, Apache need only be enabled for the duration of the certificate generation and subsequent renewals, saving server resources and reducing your security exposure.

On a default install of Apache there is probably not a lot required to be configured. Apache comes with a default site for both SSL and HTTP. In the /etc/apache2/sites-available/ one should see a "default" and "default-ssl" file.

Check that mod_ssl is enabled:

$ sudo a2enmod mod_ssl

 Check whether the "default-ssl" and "default" sites are enabled:

$ ls -la /etc/apache2/sites-enabled/
-rw-r--r-- 1 root root   692 Dec  3  2015 default -> ../sites-available/default
-rw-r--r-- 1 root root  7251 Dec  3  2015 default-ssl -> ../sites-available/default-ssl

You don't need the default site, because you shouldn't be serving up other pages for sport, only webmail over SSL (HTTPS). 

$ sudo a2ensite default-ssl
$ sudo a2dissite default

Ideally you would copy those sites and customise your own sites based on those files, for example: 

$ ls -la /etc/apache2/sites-enabled/
lrwxrwxrwx 1 root root   47 Jun  1 16:45 ssl-mail.moff.tech.conf -> ../sites-available/ssl-mail.moff.tech.conf

Notice that when you use the a2ensite / a2dissite a symbolic link is created from sites-enabled to sites-available.

We'll discuss now the customisation of the ssl site file, including the stanza for squirrelmail. Please bear in mind the following:

• This is an abridged version to highlight the things that need to be changed.
• I'm not promising you the most secure or optimised configuration. 
• It's really up to you to understand and research the settings here via Apache's useful, although sometimes dense, documentation. For security, you might want to start here.

<VirtualHost *:443>
        ServerName mail.moff.tech:443
        DocumentRoot "/usr/share/squirrelmail"
        ServerAdmin me@example.com
        SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
        SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
</VirtualHost>

I'm using example.com because I don't want spam robots harvesting my email address from this blog post.

We've altered the document root to be the squirrelmail install directory. If squirrelmail isn't installed yet, then this is going to cause Apache to complain when reading the config. At the very least, install squirrelmail (we'll customise it later) and set the document root accordingly. Note that this should not be set to the /etc/squirrelmail directory.

If you don't want to use squirrelmail, just leave DocumentRoot as default.

We'll revisit the SSL certs after sorting out letsencrypt. For now, leave them at the defaults (self signed certs), whatever they be called.

Don't move on to the letsencrypt stage until you can get Apache to start your ssl site successfully and have checked that your mail web address resolves and serves some web page - albeit with a self signed cert warning from your browser.

Let's Encrypt and Apache Integration

I don't need to go into a lot of detail here for setting up a Let's Encrypt certificate. Let's Encrypt have issued a million certificates and there's about as many user howtos and blogs to step you through. It works remarkably well as long as you do everything to the letter and have a simple Apache ssl site setup, like ours.

In truth you can use the letsencrypt standalone option and skip Apache altogether.

I've used the letsencrypt.sh client and it works well enough. I see that Let's Encrypt are now pushing something called certbot. I guess that's an experiment for me to do another time (and blog post).

Update: As of January, I notice that the letsencrypt-auto now presses certbot-auto. When running an update you receive "You are running with an old copy of letsencrypt-auto that does not receive updates, and is less reliable than more recent versions. We recommend upgrading to the latest certbot-auto script, or using native OS packages." However the cert is still succesfully issued.

Once you've stepped through the setup, check the location of the cert files. This is probably going to vary depending on the client you used to get the cert issued.

root@mail:/etc/apache2/ssl# ls -la /etc/letsencrypt/live/mail.moff.tech/
total 8
drwxr-xr-x 2 root root 4096 Jun 11 10:14 .
drwx------ 3 root root 4096 Jun 11 10:14 ..
lrwxrwxrwx 1 root root   43 Jun 11 10:14 cert.pem -> ../../archive/mail.moff.tech/cert1.pem
lrwxrwxrwx 1 root root   44 Jun 11 10:14 chain.pem -> ../../archive/mail.moff.tech/chain1.pem
lrwxrwxrwx 1 root root   48 Jun 11 10:14 fullchain.pem -> ../../archive/mail.moff.tech/fullchain1.pem
lrwxrwxrwx 1 root root   46 Jun 11 10:14 privkey.pem -> ../../archive/mail.moff.tech/privkey1.pem

That confirmed, you can head back to the Apache ssl site and verify the certificates. Where you had this:

SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

You should now have (or add) this:

SSLCertificateFile      /etc/letsencrypt/live/mail.moff.tech/cert.pem
SSLCertificateKeyFile   /etc/letsencrypt/live/mail.moff.tech/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/mail.moff.tech/chain.pem

Reload (you don't need to restart) Apache and check for errors at startup. Now you can disable Apache if you don't want to set up webmail.

The certificate you've been issued only lives for a few months. Consider a crontatb entry to renew the certificate. For example, this renews every three months (months 3, 6, 9, 12) on the 8th day:

1 0 29 JAN,APR,JUL,OCT * /root/letsencrypt/letsencrypt-auto renew

I have a script (tested and verified with crontab) to handle the renewal process because, as we will see shortly, when the certificates are required for multiple jobs the file permissions don't suit every daemon. Check out letsencrypt-handle-certs.sh on my gitlab repository.

Finally, when Apache is enabled, verify your SSL/TLS strength via SSL Labs.
 

Squirrelmail

The default install of sqm offers an apache.conf but I recommend that you ignore it. It doesn't show you the HTTPS context and you don't want to expose configtest.php anywhere.

There is a lovely script to configure Squirrelmail just as you want it:

$ sudo squirrelmail-configure

Some of the config points are these:

$imapServerAddress      = 'mail.moff.tech';
$imapPort               = 993;
$useSendmail            = false;
$smtpServerAddress      = 'mail.moff.tech';
$smtpPort               = 25;

Troubleshooting squirrelmail 

When sending an email via Squirrelmail, it may fail with a 550 message about relay not being permitted. This is because Squirrelmail didn't pass AUTH so the server doesn't trust the client and won't relay for it. The quick resolution is to add the server IP into the list of "Machines to relay mail for" in the Exim config as I mentioned in part 2.

I found that Squirrelmail will try an AUTH before Exim offers it. In other words, if you set this in /etc/squirrelmail/config.php:

$smtp_auth_mech = 'login';

Then the result may be a new problem:

Bad sequence of commands
503 AUTH command used when not advertised

Exim wants to do AUTH after TLS but Squirrelmail tries to do AUTH before that. The workaround for this is to set $smtp_auth_mech = 'none'; and make sure that the IP of the server is set as a host to be a smarthost for. This setup of Squirrelmail doesn't use TLS to connect to Exim, I can't make that work with AUTH. I ended up with a message about being unable to open the TCP stream and this in the syslog:

SMTP protocol synchronization error (input sent without waiting for greeting): rejected connection from H=mail.moff.tech [64.x.x.x] input="\026\003\001"

Since packets from Squirrelmail never leave the box, from a security standpoint there's little to gain with a TLS connection - unless you distrust your server processes or users. Some web searching might reveal a solution to setting up Squirrelmail with TLS and AUTH. If you find one, let me know!


Squirrelmail will use the system user credentials by default. If you try to login as root, then squirrelmail will fail to get in. You'll see something like this in the syslog:

Jun 22 22:57:07 mail dovecot: imap-login: Login: user=<root>, method=PLAIN, rip=x.x.x.x, lip=y.y.y.y, mpid=25745, TLS, session=<N5s1LuQ1HQBAFlV3>
Jun 22 22:57:07 mail dovecot: imap: Error: user root: Invalid settings in userdb: userdb returned 0 as uid
Jun 22 22:57:07 mail dovecot: imap: Error: Invalid user settings. Refer to server log for more information.

Setting the servername to match the one on the certificate will avoid errors such as this:

Jun 11 12:06:05 mail dovecot: imap-login: Disconnected (no auth attempts in 0 secs): user=<>, rip=::1, lip=::1, TLS, session=<Tvayzf00ZQAAAAAAAAAAAAAAAAAAAAAB>

Above, the certificate servername validation was probably failing. You can't use the loopback address or you'll get that kind of error.

Additionally, an error log message like this one:

Jun 11 11:55:55 mail dovecot: imap-login: Disconnected (no auth attempts in 0 secs): user=<>, rip=x.x.x.x, lip=y.y.y.y, TLS: SSL_read() failed: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca: SSL alert number 48, session=<qutPqf00mABSh18U>

Will likely happen if the CA is unknown. You might see this if you've copied a self signed cert from another server without copying the "Self CA" as well. Rather than faff about sorting out a self signed cert, use Let's Encrypt.

Dovecot Certificate Integration

To splice the new certs into the Dovecot configuration, you might get away with simply calling the Let's Encrypt cert files where they stand.

Edit /etc/dovecot/dovecot.conf:

ssl_cert = </etc/letsencrypt/live/mail.moff.tech/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.moff.tech/privkey.pem

Then reload and watch the logs for errors.

After my first attempt at adding the Let's Encrypt cert, Thunderbird reported that my cert was still essentially self signed. I hadn't used the "full chain" file so apparently that meant Thunderbird wasn't too sure who the root CA was. This is possibly because Thunderbird didn't have the intermediate CA certs stored. Mozilla applications will apparently tuck a received Intermediary CA cert away for later reference. Using the full chain file allows Thunderbird to iterate the chain of authority.

That's also why you set the chain file in the Apache config.

Exim4 Certificate Integration

The exim4 config file states categorically:

# Full paths to Certificate and Private Key. The Private Key file
# must be kept 'secret' and should be owned by root.Debian-exim mode
# 640 (-rw-r-----). exim-gencert takes care of these prerequisites.

As it's a bad idea to tinker with the cert file permissions and owner/group. Copy them into the /etc/exim4/ directory and set the permissions accordingly:

# ls -l /etc/exim4/ | grep 'pem'
-rw-r----- 1 root Debian-exim  3456 Jun 12 22:01 fullchain.pem
-rw-r----- 1 root Debian-exim  1704 Jun 12 22:01 privkey.pem

If you're copying over the old files, then you don't need to change the exim4 default configuration. If you're editing the exim config, here's how:

/etc/exim4/conf.d/main/03_exim4-config_tlsoptions:

.ifndef MAIN_TLS_CERTIFICATE
#MAIN_TLS_CERTIFICATE = CONFDIR/exim.crt
MAIN_TLS_CERTIFICATE = CONFDIR/fullchain.pem

.ifndef MAIN_TLS_PRIVATEKEY
#MAIN_TLS_PRIVATEKEY = CONFDIR/exim.key
MAIN_TLS_PRIVATEKEY = CONFDIR/privkey.pem

Notice that we're using fullchain again.

I'm handling these steps in a cert renewal script, but at the time of writing I'm not ready to publish that script due to it having had very little testing.

After updating the cert file location, reload exim.

# service exim4   reload

Watch /var/log/syslog, /var/log/exim4/mainlog and /var/log/exim4/rejectlog for unusual messages.