Https with PerfSonar 4.3.4

Whats this about

Congratulations you have a perfSonar instance setup and running.

If you don't then please read one of the many ancient repositories of sysadmin knowledge distributed across the explored region of the local galactic cluster, and please remember, don't panic.

In Edinburgh at the time of writing we have perfSONAR 4.3.4 running as installed from the official perfSONAR iso images on CentOS7 deployed in February 2020. https://docs.perfsonar.net/install_getting.html

This guide is written to explain/educate so I apologise if some of these steps are overly verbose and/or covering ground well understood by people familiar with most/all of the tools written here.

The https configuration in perfSONAR

perfSONAR supports using https for providing access to several internal pages including secure web-pages for the per-server configuration.

The important file covering this configuration is:

/etc/httpd/conf.d/ssl.conf

As part of the installation perfSONAR generates a self-signed certificate for https connections. This is not best practice but not intrinsically insecure if you chose to trust this certificate properly.

However this certificate (by default) is only valid for 12 months. This means that 1-year after your installation you should generate a new certificate. There is no component within perfSONAR which checks and updates this certificate every year.

Speaking to at least 2 sysadmins and after checking multiple sites it's clear that the use of this limited/short-lived certificate is not well advertised and so I've written this guide to replace it.

Getting a letsencrypt certificate

As of 2021 Python2.x is dead long live Python3. Instructions here are very Python3 dependent. Trying to do the equivalent in Python2.x is just not advised anymore.

First step install the virtualenv tool on your perfsonar box.

# Install epel
yum install epel-release
# Install python3.6 virtualenv from epel
yum install python36-virtualenv

Now build a virtualenv for certbot to run in and install certbot there (NB: yes I am running this as root on purpose!):

# Build a Python3 virtual environment so as to not pollute the host
virtualenv-3 /root/certbot_env
# Source/Activate this virtual env so that we can install Python packages
source /root/certbot_env/bin/activate
# Install the most recent certbot client in this env
pip install certbot

To check this is all setup correctly:

# which certbot
/root/certbot_env/bin/certbot

Now in order to use certbot with letsencrypt you will need access to privileged ports to respond in a secure way to confirm that you are in control of the server which you're wanting a certificate to be issued for (challenge-response).

In order to make proper use of these ports we should first stop the apache service running. Then we can get the certbot tool to do what it does for us:

# Stop apache from perfSONAR (temporary)
systemctl stop apache
#
# Use the certbot tool to request certs be issued by letsencrypt for our sever in an isolated way:
# For Edinburgh the 2 hostnames which point to the 1 box are:  gridpp-ps-band and gridpp-ps-lat
# We want the service to be secure when accessed via either interface.
#
certbot -d gridpp-ps-lat.ecdf.ed.ac.uk -d gridpp-ps-band.ecdf.ed.ac.uk certonly --standalone

Please follow the appropriate prompts and give a sensible email address so that you get notified when the cert is next due to expire.

You can now restart the perfSONAR apache instance if we want, but it's not any more secure (yet!)

If all went well you should now have a folder under

/etc/letsencrypt/live/

with the first FQDN you requested above. In the case of Edinburgh this folder was:

/etc/letsencrypt/live/gridpp-ps-lat.ecdf.ed.ac.uk

.

This folder should contain something like the following:

# ls /etc/letsencrypt/live/gridpp-ps-lat.ecdf.ed.ac.uk/
cert.pem  chain.pem  fullchain.pem  privkey.pem  README

Installing a new certificate for apache

The default perfSONAR configuration makes use of a certificate and private key for the host stored at:

# Certificate 
/etc/pki/tls/certs/localhost.crt
# Private Key
/etc/pki/tls/private/localhost.key

These files are referenced from the apache ssl configuration:

/etc/httpd/conf.d/ssl.conf

In order to use the files as generated by letsencrypt the (best?) solution is to re-configure apache to point to the new letsencrypt certificates. To do this the ssl.conf file should be edited such that the following section looks similar to below:

...

#SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateFile /etc/letsencrypt/live/gridpp-ps-lat.ecdf.ed.ac.uk/fullchain.pem

...

#SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
SSLCertificateKeyFile /etc/letsencrypt/live/gridpp-ps-lat.ecdf.ed.ac.uk/privkey.pem

...

#SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt
SSLCertificateChainFile /etc/letsencrypt/live/gridpp-ps-lat.ecdf.ed.ac.uk/chain.pem

...

i.e. localhost.crt == fullchain.pem, localhost.key == privkey.pem and it's good to let apache know about chain.pem

NB: If you want to check file permissions be aware that certbot/letsencrypt use symlinks by default and you will need to follow them to the origin to check the ultimate file permissions which are presented to apache when it reads a given file.

After this you should be free to (re-)start apache again for perfSONAR:

systemctl start httpd

You should now see a secure padlock (or equivalent) when you point your bowser to your perfsonar instance web-interface, e.g. for Edinburgh https://gridpp-ps-band.ecdf.ed.ac.uk/toolkit/

Renewing a letsencrypt certificate

To be written but again can be done using the certbot above. The process is likely the same as above but including a renew command to certbot.

Updating the certbot instance (if/when needed) should be as easy as:

pip install --upgrade certbot

within the virtualenv setup earlier.

The Dirty Durham Way

Working from ECDF's findings above, we've push Lets Encrypt to PerfSonar is a very dirty (read unstable) fashion. This assumes you're running the perfsonar image based on CentOS.

Steps:

1. Install Perfsonar
2. Install certbot
3. Modify /etc/httpd/conf.d/ssl.conf
4. restart apache
5. Request LE cert
6. Modify /etc/httpd/conf.d/ssl.conf
7. Restart apache

Install Certbot

yum install certbot

or

pip install certbot

Edit SSL Settings

Add this line to your /etc/httpd/conf.d/ssl.conf (you can put it anywhere but this file makes sense to me) just before the closing virtual host tag

AliasMatch "^.well-known" "/var/www/html/.well-known"

then reload/restart apache

systemctl reload httpd

Request certificate

Request a certificate via the certonly and webroot option. such as below, change your hostname to your public hostname

certbot certonly --webroot -w /var/www/html/ -d perfsonar.dur.scotgrid.ac.uk

or

certbot certonly --webroot -w /var/www/html/ -d $(hostname -f)

Re-Edit SSL Settings

You then need to re-edit the /etc/httpd/conf.d/ssl.conf, you need to edit

1. Line 100 - SSLCertificateFile /etc/pki/tls/certs/localhost.crt
2. Line 107 - SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
3. Line 116 - SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt

To the following (change the location to your domain):

1. SSLCertificateFile /etc/letsencrypt/live/perfsonar.dur.scotgrid.ac.uk/cert.pem
2. SSLCertificateKeyFile /etc/letsencrypt/live/perfsonar.dur.scotgrid.ac.uk/privkey.pem
3. SSLCertificateChainFile /etc/letsencrypt/live/perfsonar.dur.scotgrid.ac.uk/fullchain.pem

Then simply restart httpd.