Minikube and Let’s Encrypt — Renewal

Renew an expiring Let’s Encrypt certificate with Cert-Manager

Photo by K. Mitch Hodge on Unsplash

Two months ago I published an article about how to generate a Let’s Encrypt certificate for Minikube using cert-manager. One of the features was that you would generate a backup of the certificate and re-install it easily when Minikube gets restarted or your cluster died for some reason.

Now it’s time to renew the certificate: Let’s Encrypt certificates last 90 days, and there is a 30-day window where the cert-manager is going to renew the certificate automatically. So as long as the Minikube cluster is up and running, you don’t need to worry, but here is how to create a backup of that renewed certificate.

This article is closely related to that earlier article — but as that earlier article is already lengthy, I wrote this new section separately instead of expanding the original.

TL;DR

You can use a simple kubectl command to extract Kubernetes secrets by using the -o yaml option. These yaml files can be stored and re-applied. This is shown here, plus some introduction to the kubectl plugin for cert-manager that helps to handle Let’s Encrypt certificates.

What you need to bring

For the practical experience, you will need to set up a Minikube cluster according to the original article, mainly because of some conventions such as file names. For the remainder of this follow-up article, it is expected that you have a running Minikube cluster with the cert-manager and a production certificate installed, following the instructions.

If you have a production certificate, but Minikube is not running anymore… you can restart the cluster and install the existing certificate by just calling the ./start.sh script: it will detect the certificate and install it along cert-manager.

How to know about expired certificates

There are many ways to figure out if a certificate is valid. For example, you can use a popular browser: navigate to your site, click on the lock-icon next to the URL and click on a few other buttons depending on your browser to eventually see the certificate details, including the validity.

If you are wondering about the validity of the TLS secret backup that we generated before, you can use the check-cert.sh script that is now available with the original GitHub repository. As always, it is a good idea to review any script that you get from somewhere before running it locally. Doing so will show that we use a tool called yq for YAML conversion to JSON and jq for JSON parsing. You will need to install these (see script sources for details).

./check-cert.sh prod-my.domain.com-cert.yaml

You need to use your backup file name, obviously, and something like this may come out:

Certificate prod-my.domain.com-cert.yaml is valid until 2021-07-05

If the date is less than 30 days away, it is time to renew. If you have a running cluster, cert-manager may have renewed the certificate already, but that did not update the backup. As production certificate handling is rate-controlled by Let’s Encrypt it is useful to produce an updated backup after a renewal.

kubectl cert-manager plugin

This section is optional, but the cert-manager's kubectl plugin gives you a lot of insight: install the plugin according to instructions, and call it with KUBECONFIG set as needed for your cluster. Here we are using the default namespace from the original setup:

export KUBECONFIG=kubectl.config
kubectl cert-manager status certificate my-app-tls -n my-app

You will see a lot of information, and at the very end, there is a Not Before and Not After section, describing the currently installed certificate validity time, which may be different from the backup if cert-manager already updated the certificate.

Not Before: 2021-06-16T13:37:04-05:00
Not After: 2021-09-14T13:37:03-05:00

In this case, at the time of writing, the backup is still valid, but renewal happened — therefore we see different validity dates in the live cluster.

The cert-manager plug-in can do a lot more things, including triggering a renewal. So far, this was not needed for me as cert-manager renewed automatically. So here is a big Thank-You to the cert-manager team for their work. It just works.

Extract a new backup certificate file

The simple kubectl command to generate a new backup is this:

kubectl -n my-app get -o yaml secret my-app-tls > prod-my.domain.com-cert.yaml 

This overwrites your existing backup (so you may want to backup the backup before running the command or use other file names). Once you have done the above, you should be able to restart the cluster using ./start.sh when needed.

That’s it… not complicated, but I hope this helps at least one of you out there :-) Thanks for reading.

One more thing…

The above is a pretty manual process — I am sure that there is an automated way to generate a certificate backup. Let us all know in the comments.

Finally, there is another script called renew-cert.shin the GitHub repository that does not require the cert-manager plugin to check the active certificate, and that polls the cluster for a renewal. This may be useful for some. It’s not a pretty script but it shows how to extract dates from both the backup file and the running cluster.