Savannah TLS Information

Savannah has been using Let's Encrypt certificates since around 2016. Prior to that time a number of different certificate authorities were used. But the EFF's Let's Encrypt project has become very mainstream. Savannah systems have been using the EFF's Let's Encrypt certificates since then.

Certificate Expiration Event September 2020

On September 30, 2021, as planned the DST Root CA X3 cross-sign has expired for the Let's Encrypt trust chain. That was a normal and planned event. However coupled with a verification error in the code of libraries authenticating certificates it caused some clients that have not updated to fixed versions to have problems validating certificates.

If you are experiencing invalid certificate chain problems with any of Savannah's Let's Encrypt certificates then please upgrade your client to the latest patches for your system. There are too many systems to track and it would be a duplication of other efforts to track those here. Instead please reference these resources as to upstream information and discussion about the issue.

  • https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
  • https://community.letsencrypt.org/t/production-chain-changes/150739/4
  • https://letsencrypt.org/docs/certificate-compatibility/
  • https://letsencrypt.org/certificates/
  • https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/

System Updates

Of course any system operating with security support should have already updated at least three packages.

  • gnutls
  • libssl
  • ca-certificates

The first two (gnutls and libssl) control the validation of the trust path. The original versions operated incorrectly in validating the trust path in a way that is triggered when the default trust path has an expired certificate but an alternative path exists that is fully valid. Because the libraries fail if any of the paths fail, not succeed if any of the paths succeed. To paraphrase and simplify the description. See the above references for the deep dive into the technical details.

The last one (ca-certificates) contains the trusted root certificates used to anchor the trust path. Routinely trust path anchors change as the Internet is always in motion and always changing. It is important to have a current set of trust anchors.

Reported Client Side Problems

There have been user reports of problems with SSL/TLS connections with the certificate validation. It was asked that we hint here to users experiencing those problems so that they will know it is a client side issue and not a server side issue. I will give an example system that is problematic by default without any automatic upgrade available.

  • Trisquel GNU/Linux Belenos 7

This is an example of a system that is out of security support. By default it will have trouble validating the trust path now that the DST Root CA X3 has expired.

Example of a wget https certificate validation failure due to errors validating due to expired DST Root CA X3.

$ wget -S -O/dev/null https://savannah.gnu.org/
--2021-10-03 18:37:25--  https://savannah.gnu.org/
Resolving savannah.gnu.org (savannah.gnu.org)... 2001:470:142::72, 209.51.188.72
Connecting to savannah.gnu.org (savannah.gnu.org)|2001:470:142::72|:443... connected.
ERROR: cannot verify savannah.gnu.org's certificate, issued by ‘/C=US/O=Let's Encrypt/CN=R3’:
  Issued certificate has expired.
To connect to savannah.gnu.org insecurely, use `--no-check-certificate'.

Example of a curl https certificate validation failure due to errors validating due to expired DST Root CA X3.

$ curl -LD- -o/dev/null https://savannah.gnu.org/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (60) SSL certificate problem: certificate has expired
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Because Trisquel 7 is out of security support a simple upgrade from the software distribution repositories will not automatically upgrade the packages needed to mitigate this problem. One would need to manually install a newer ca-certificates package and/or build backported gnutls and libssl libraries.

Fortunately a simpler mitigation is possible. Commenting out the DST Root CA X3 certificate from the ca-certificates.conf file and rebuilding the database is minimally sufficient. Locate the "mozilla/DST_Root_CA_X3.crt" in the /etc/ca-certificates.conf file, comment it out using a "!" at the beginning of the line, run update-ca-certificates to rebuild the database, then test the result.

Here are a few useful command line commands. First we can grep to check the status of that line in the file. Is it there? Is it active? Is it commented out?

grep DST_Root_CA_X3 /etc/ca-certificates.conf

If it is there and not commented out then we can use this one line sed command to edit the file. And then follow with the update-ca-certificates to rebuild the database.

sed --in-place '/DST_Root_CA_X3.crt/s/^/!/' /etc/ca-certificates.conf
update-ca-certificates

Then test the result.

wget -q -S -O/dev/null https://savannah.gnu.org/
curl -sLD- -o/dev/null https://savannah.gnu.org/

Even Older Systems

Even older systems will need an updated ca-certificates anchor store as they will be so old they do not know about the newer ISRG Root X1 certificate anchoring the current trust path. This can be manually accomplished by downloading the ISRG Root X1 certificate in pem format from https://letsencrypt.org/certificates/ and installing it in /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt, adding it to the /etc/ca-certificates.conf file, running update-ca-certificates to rebuild the database, and then certificates should be trusted again.

  • https://letsencrypt.org/certs/isrg-root-x2-cross-signed.pem

I would like to say to "wget"/"curl" the above into place but being an https link that likely will not work. One can add the "curl --insecure" and "wget --no-check-certificate" options. But obviously those are not secure ways. Copy to the local desktop clipboard and then pasting into the file from the clipboard may be good enough.

Add the certificate to the /etc/ca-certificates.conf anchors and then run update-ca-certificates to update the database. Then test the result.