In my post on container lifetimes I discussed how I wanted to make keycloak perform dynamic SSL certificate reloading. The maximum lifetime of an SSL certificate is no longer dictated by the Certificate Authority, but rather a cabal of web browser developers who wield a big schwartz. I am using free certificates provided by Letsencrypt, you may think that is amateur or juvenile, but I think it’s extortion what CAs charge for a certificate.
Keycloak is built on top of the WildFly (JBOSS) Java application server (which coincidentally is not vulnerable to CVE-2021-44228) and thus it relies on the underlying JBOSS infrastructure for handling SSL certificates and keystores. I am using the Docker container built by the keycloak team as the basis for my authentication solution. The Docker container relies on SSL certificates being present at the path /etc/x509/https/cert.(pem|key) during container initialization. If the files are present then the script /opt/jboss/tools/x509.sh is run to import the PEM format certificates into the Java keystore that the application server uses.
The specific subsystem that keycloak uses for managing the SSL certificates is named elytron, and thus this is the subsystem that must be referenced when reloading the certificates. To successfully reload SSL certificates are runtime, you need to do these things:
- Update the SSL certificates at /etc/x509/https/cert.(pem|key)
- Recreate the Java keystore by calling
/opt/jboss/tools/x509.sh - Execute a JBOSS CLI script to load and initialize the keystore
Keycloak doesn’t offer a canned solution for the above procedure, so you need to create some glue logic to do this. My solution was to create a daemon called notify_watcher that uses the inotify API to watch a git repo which is mounted on a docker volume. When the certificate directory in the git repo is updated then notify_watcher calls a shell script which performs the above steps.
It is necessary that your automation run inside the keycloak container because the admin port for the JBOSS server is only listening on localhost, and I think it is wise to leave this as it is. My solution is to build a base container for notify_watcher, then create a Dockerfile with multiple FROM references so I can copy the baked Python script out of the Docker image, then run some commands to install Python and a venv in the Keycloak image. This method is a lightweight solution for creating a derivative container without having to maintain messy references to notify_watcher in the git source repo — just build the image and copy from a well known location.
Once I have notify_watcher in the container and the glue logic, the process is relatively painless. The notify_watcher program has a random hold-off capability so if there are multiple instances running in a stack they don’t all execute their tasks at the same time.
The magic incantation you feed to jboss-cli.sh to cause keycloak to reload the keystore is very simple:
/subsystem=elytron/key-store=kcKeyStore:load() /subsystem=elytron/key-manager=kcKeyManager:init()
Once that stanza executes, keycloak will begin serving the new SSL certificate.
