How to get the SSL/TLS certificate chain right

After installing Firefox and Chrome on a new PC, I noticed that I was getting "issuer unknown" (Firefox: SEC_ERROR_UNKNOWN_ISSUER) errors on a website that I was checking connectivity against. The website was one that I was recently put in charge of and the organisation had a paid-for COMODO certificate for a Wordpress install.

On other computers there were no such errors reported by Firefox or Chrome for this website, so I initially missed the significance of the problem.

Background

Note: "Certificate" should be read as "public certificate". The private cert or key is not discussed in this document.
 
Websites that support HTTPS require a valid SSL/TLS certificate or the client will receive certificate warnings from the application or browser. Happily, your community organisation or personal website can get by with a free certificate courtesy of Let's Encrypt. At the same time, the website needs to provide a certificate chain, which essentially informs the client (your browser) about the identity of the host that signed your certificate.

Likewise, the certificate of the host that signed the previous host's certificate needs to be provided. This recursive method of providing the certificate of the previous "intermediary" signer continues until the root certificate authority (CA) is reached. The root CA certificate does not need to be provided because it should be specifically trusted on the client software or browser or the whole infrastructure of trust is useless. Browsers and other software will ship with root CA certificates, or you can manually add them if necessary.

Any intermediary CA needs to be included in the certificate chain, but the root CA should not be included.

Incomplete, Contains anchor

I turned to Qualys SSL Labs to see whether I could obtain a head start on the problem. I saw warnings, which told me where to look but didn't help in identifying exactly what was wrong.

"This server's certificate chain is incomplete. Grade capped to B."

And later in the report:

"Chain issues - Incomplete, Contains anchor"

 I had somewhere to investigate at least - the certificate chain.

Also worth noting was that curl also complained of certificate problems on my desktop even if Firefox and Chrome did not.

Examining the Certificate Chain

Note: example.com is a placeholder for the real URL I was investigating.
 
openssl is the obvious tool to turn to for seeing the nitty-gritty of an SSL/TLS session. I was able to easily view the certificate chain:

$ openssl s_client -connect www.example.com:443
...
Certificate chain

 0 s:/OU=Domain Control Validated/OU=Hosted by webgo GmbH/OU=PositiveSSL/CN=www.example.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA

 1 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root

 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root

To the untrained eye, the above output looks incomprehensible. But with a little understanding and research the problem can be clearly seen. The chain consists of three certificates (0,1,2) issued by the website. Each certificate has a (s)erver that the certificate belongs to and an (i)ssuer that signed the certificate for that server.

The first certificate (0) is for www.example.com and it was issued by COMODO RSA Domain Validation Secure Server CA. 

The second certificate (1) is expected to be for the COMODO issuer in the first certificate, but it is not. The certificate is for some other server AddTrust External CA Root, who is oddly also the issuer (a so called "self signed" certificate). Things are broken to bits from this point.

The third certificate (2) is completely superfluous because the second cert in this chain should not be there. In this third cert, we see it is for the server COMODO RSA Certification Authority and has been signed by the issuer AddTrust External CA Root.

Fixing the Mess

The solution was to provide a correct certificate chain. The first certificate (our certificate) was valid, but since it was signed by "COMODO RSA Domain Validation Secure Server CA" that needs to be the next public certificate found in the chain.

I first checked whether this was part of a typical Firefox CA set. The shipped CA certs can be viewed either on the Mozilla website or via the options->preferences of Firefox itself:

Yeah that's in German, sorry, but the English version will look basically the same.

In this case, notice that in the above screenshot "COMODO RSA Domain Validation Secure Server CA"  is actually in my certificate list in Firefox. I realised afterwards that at some point I had clicked through the invalid certificate warnings and added the certificate to my Firefox certificate store, to be trusted for next time. That's why I only noticed the problem after installing a new PC with Firefox and Chrome.

Just to repeat myself, the "COMODO RSA Domain Validation Secure Server CA" certificate is not part of the default suite of certificates trusted by Firefox. I needed to download the public certificate from COMODO here and tell apache2 (the site's web server) to use that certificate and only that certificate as part of the certificate chain.

Briefly, this meant configuring these apache2 settings ...

SSLCertificateKeyFile /etc/apache2/ssl.key/www.example.com.key
SSLCertificateFile /etc/apache2/ssl.crt/www.example.com.crt
SSLCertificateChainFile /etc/apache2/ssl.ca/www.example.com.ca

... ensuring that SSLCertificateKeyFile  contained only the private key of the server, SSLCertificateFile contained only the public certificate associated with the aforementioned private key and that SSLCertificateChainFile contained only the public certificate for "COMODO RSA Domain Validation Secure Server CA". If you have multiple layers of signing, you need to add each intermediary CA to this file, in the correct order. 

This works because the certificate for the intermediary "COMODO RSA Domain Validation Secure Server CA" is issued by "COMODO RSA Certification Authority" and that root CA is part of the shipped set of certificates for Firefox, Chrome, curl, openssl and any other SSL/TLS client you care to name.

The certificate chain now looks like this, (see earlier openssl command syntax):

Certificate chain

 0 s:/OU=Domain Control Validated/OU=Hosted by webgo GmbH/OU=PositiveSSL/CN=www.example.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
 
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority

In the first certificate, the (i)ssuer name is the (s)erver name of the second certificate. The second validates the first. There is no need for a third certificate because the issuer of the second certificate is part of an established set of known root Certificate Authorities.