# How we recovered a k3s cluster after its client certs expired

119 ephemeral preview namespaces should have been reaped between 2am and 6am. None were. The teardown cron had been failing on x509: certificate has expired or not yet valid for six hours before anyone looked at the log. The k3s server was 366 days old, nobody owned certificate rotation, and the first fix we tried made zero difference because we misread which cert was expired. This is how we got kubectl working again, in what order we restarted the control plane, and the two things we changed so the next cluster does not do this to us.

**Problem signals:**

- kubectl get nodes returns 'Unable to connect to the server: x509: certificate has expired or not yet valid'
- A CI or lifecycle script that used to work is now failing on every kubectl call, and the cluster has been up more than 11 months
- systemctl status k3s shows the service is active but the k3s.service journal has TLS handshake errors from the agent
- The kubeconfig at /etc/rancher/k3s/k3s.yaml decodes to a client cert whose notAfter is in the past
- Ephemeral or preview environments are stuck: nothing is being created, nothing is being deleted, and the queue is growing

## What we walked into

*119 namespaces that should have been gone by 6am*

The cluster ran preview environments for a mid-size SaaS product. Every merged PR got a namespace, every closed PR got its namespace reaped by a cron that shelled out to kubectl. Simple, effective, and completely dependent on kubectl being able to talk to the API server. On the morning of the incident the reap job had been erroring every minute for about six hours before the platform on-call noticed the namespace count on the Grafana panel had gone flat instead of sawtoothing.

The error in the cron log was one line, repeated:

```
Unable to connect to the server: x509: certificate has expired or not yet valid: current time 2025-04-12T06:14:22Z is after 2025-04-11T18:02:17Z
```

*The API server had been unreachable for twelve hours and twelve minutes by the time we opened the log.*

This is a single-server k3s install with one agent node, running on two small EC2 instances. It had been up for 366 days, one day longer than its own certificates were valid for. Nobody had ever rotated a certificate on it. Nobody had written down that k3s issues its own internal client certs with a 12 month lifetime and quietly renews them on restart if they are within 90 days of expiry. Ours were nowhere near that window when the server last restarted, because the last restart was the kernel patch cycle that brought this cluster up in the first place, twelve months earlier. So they aged out on schedule and the cluster locked itself out.

## The first thing we tried, and why it did not work

*The kubeconfig regen that fixed nothing*

The instinct on the first responder was correct-looking and wrong. They assumed the kubeconfig on the CI runner had drifted, copied a fresh /etc/rancher/k3s/k3s.yaml from the server, rewrote the server URL, and reran the reap. Same x509 error. They tried it a second time with KUBECONFIG pointed explicitly at the new file. Same error.

The reason it did not work is the piece that catches people out with k3s specifically. The k3s.yaml on disk is not a pointer to cert files; it is a self-contained kubeconfig with client-certificate-data and client-key-data base64-embedded directly in it. Regenerating the file by copying it from the server just copies the same expired bytes to a new path. The cert material lives at /var/lib/rancher/k3s/server/tls/client-admin.crt, and that is what had actually expired. Until that file was rotated, every kubeconfig on every machine, freshly copied or not, was carrying the same dead cert.

This is the point where we stopped guessing and asked openssl what it saw.

## How we confirmed the actual expiry before touching anything

*openssl said notAfter=Apr 11*

Before running any rotation command we wanted to see the cert dates ourselves. Two commands, run on the k3s server:

```
$ sudo openssl x509 -noout -dates -in /var/lib/rancher/k3s/server/tls/client-admin.crt
notBefore=Apr 11 18:02:17 2024 GMT
notAfter=Apr 11 18:02:17 2025 GMT

$ for f in /var/lib/rancher/k3s/server/tls/client-*.crt /var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt; do
>   echo "$f"
>   sudo openssl x509 -noout -enddate -in "$f"
> done
```

*The client-admin cert expired at 18:02 the previous evening. Every other client cert on the server had the same notAfter within a few seconds.*

The serving cert for the API server itself was also expired. That mattered for the restart order later, because a k3s server with an expired serving cert will start, but agents cannot re-establish TLS to it until the rotation writes new material and the server picks it up. If we had restarted the agent first, or in parallel, we would have watched it fail to rejoin and then chased a second ghost.

Related reading: the same failure mode shows up during cluster migrations when a snapshot from an old server gets restored past the cert lifetime. We wrote about that pattern in [the migration recovery notes](https://infraforge.agency/migrations/).

## The recovery sequence that actually worked

*Rotate, restart server, restart agent, in that order*

k3s ships a rotation subcommand that regenerates the internal certs on disk. It does not restart the service, and it does not touch k3s.yaml; the restart in step 3 is what rewrites that file. The second trap is everything downstream of the server, which we get to below. The full sequence, run as root on the server, was:

```
# 1. Stop the server so nothing is holding the old TLS material.
sudo systemctl stop k3s

# 2. Rotate all internal certs. On k3s v1.28+ this rewrites everything
# under /var/lib/rancher/k3s/server/tls including client-admin.crt.
sudo k3s certificate rotate

# 3. Bring the server back up. It will pick up the new certs on start.
sudo systemctl start k3s

# 4. Wait until the API is responsive with the local (root-owned) kubeconfig.
sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml k3s kubectl get nodes

# 5. On the agent node, restart so it re-handshakes with the new server cert.
sudo systemctl restart k3s-agent
```

*Stop first, rotate, start, verify, then touch the agent. The k3s docs are explicit that the server has to be stopped before rotating, and we have separately seen a rotate against a live server leave the API server holding the old serving cert in memory until a restart anyway.*

Step 4 worked from root on the server. On the CI runner, kubectl was still failing. This is where the k3s.yaml quirk from earlier comes back, with the sign flipped. k3s rewrites /etc/rancher/k3s/k3s.yaml on every start, so by the time step 4 ran, the server's own kubeconfig already carried the new embedded material; that is exactly why step 4 worked. Nothing rewrites the copies. Every kubeconfig we had ever scp'd to a runner or a laptop still held the dead bytes, and no amount of restarting the server was going to reach them. We had to export the fresh one and re-seed each consumer by hand:

```
# On the server, export the kubeconfig k3s rewrote when it started.
# config view --raw only prints what is already on disk; the restart
# in step 3 is what put the fresh cert material there.
sudo k3s kubectl config view --raw > /tmp/k3s-fresh.yaml

# Verify the embedded client cert is not the dead one.
grep client-certificate-data /tmp/k3s-fresh.yaml \
  | awk '{print $2}' | base64 -d \
  | openssl x509 -noout -dates
# notAfter=Apr 12 06:41:07 2026 GMT   <-- future date, good.

# Distribute to the CI runner's kubeconfig path. In our case:
scp /tmp/k3s-fresh.yaml ci-runner:/etc/lifecycle/kube/config
ssh ci-runner 'sudo chown lifecycle:lifecycle /etc/lifecycle/kube/config && sudo chmod 600 /etc/lifecycle/kube/config'
```

*The server's own kubeconfig is correct after the restart. The copies are not, and nothing on the cluster knows they exist. Every consumer holding one has to be re-seeded by hand.*

Elapsed time from opening the log to a successful kubectl get nodes on the CI runner was 47 minutes. Most of that was the first-fix detour. The rotation itself took about 90 seconds; the server restart took under 20.

![The left branch is where we spent 30 wasted minutes. The right branch is what the runbook now says to do first.](https://kroki.io/mermaid/png/eJxVkM1u2zAQhO9-inkBBwGCHpJDgsR2_m8t0APhA02tTEIUKeyuIgt1370gkxrOeeebmZ025sl5y4pf6wVwbw4_Lq9v4Ig1tMFZJXgroMMQmBqEBMc5Ieb9FsvlLR7-_PbB-QogCKzT0cY4_yfu_i6AhyI9WpGxJ3TjjlxObdgfsTKrPMzoruRitn2EZvCYEvF2AaxqwNr81BAjWhui3ID6HTUNNadAsT2hIdtgNyvJ9hTnPLkOU1CPPFASiUdsjIuBki5t04d04ViRst63SlxeG6xoMdjU5Ecjsyj1TiNE81BqlutjvT6Z7kq-7cRZrVJRPFXF8ze-TPxl8FzPL2ZzGDIr1J-PUkRgmjgrIadPslAvlXo16yDKYTcqoWUSf86e9qs5r5V4O6vBdCqytHtK1fityt5N9dGIPZVVGhJMmTsBfRDPkyem7T-FUrvI)

*The left branch is where we spent 30 wasted minutes. The right branch is what the runbook now says to do first.*

## FAQ

*Questions the team asked when we posted the internal postmortem*

Can I just add insecure-skip-tls-verify to the kubeconfig and move on?

No. The API server is presenting an expired serving cert, but it is also validating incoming client certs. Once the client-admin cert is past notAfter, the server rejects the request during TLS with the same x509 error even if the client stopped verifying. Skipping verification on the client side does not help you here. It also leaves the agent unable to reconnect independently.

Does k3s not auto-renew certs?

It does, but only if the server restarts while the certs are inside the 90 day renewal window. If the cluster runs 12 months straight with no restart, or the last restart happened more than 90 days before expiry, the renewal window is missed and the certs age out. That is exactly what happened to us.

Do I need to restart etcd or anything else on a single-server k3s?

On single-server k3s with the default embedded SQLite or embedded etcd, systemctl restart k3s covers everything the server needs. On an HA k3s with external etcd or multiple servers, you rotate and restart one server at a time and let the API stay reachable through the others. That is a different story and worth its own runbook.

Will the pods restart when I bounce the k3s server?

No. Workload pods run under containerd and are not managed by the k3s systemd unit's lifecycle in a way that restarts them. During our recovery, application pods kept serving traffic the entire time the API was down. Only the control plane was affected. Anything using in-cluster kubeconfigs (operators, controllers) will reconnect once the API comes back.

Is the agent's own node cert also rotated?

It needs its own attention. k3s certificate rotate only rewrites files under /var/lib/rancher/k3s/server/tls on the machine you run it on; it does not reach into the agent's /var/lib/rancher/k3s/agent/. Restarting k3s-agent works because the agent cannot authenticate with its expired client cert, falls back to re-bootstrapping with the node token, and the server issues it a fresh one. That is a fallback path, not a rotation. The deterministic version is to run k3s certificate rotate on the agent node too, servers first, then agents. Either way step 5 is not optional.

## Where we help

*If your k3s cluster is past its first birthday*

The two things we changed after this incident were not glamorous. We added a nightly check that runs openssl x509 -noout -enddate against every cert under /var/lib/rancher/k3s/server/tls and every embedded client-certificate-data in every kubeconfig we own, and pages if any is under 30 days from expiry. It is a 42 line shell script and it has already caught the same class of problem on a second cluster that a client acquired through a merger. We also scheduled a monthly forced restart of the k3s server during a maintenance window. Quarterly is not enough: restarts 91 days apart land inside the last-90-days renewal window at most once per 12 month cert, and only by a day or two. Monthly gives three restarts inside the window. Neither of these prevents the failure by itself; together they close the door.

The hard part of this kind of work is not running k3s certificate rotate. It is knowing before you touch anything which cert is actually expired, what order the components have to come back in for this specific k3s version, and which files the rotation does and does not rewrite. We do this often enough on inherited k3s and k0s clusters, especially ones that came along with an acquisition or a platform-team handover, to know where the traps are on each version. If you have a cluster that is past its first birthday and nobody on the team can point to the last time certs were rotated, [book an infrastructure review](https://infraforge.agency/review/) and we will spend a 60 minute call walking the cert state with you before it fails at 3am. If you are staring at x509: certificate has expired in a log right now, say so in the request and we will be on a bridge the same day.

---

Originally published at [https://infraforge.agency/insights/k3s-certificate-expiration-recovery/](https://infraforge.agency/insights/k3s-certificate-expiration-recovery/).

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — [see /review](https://infraforge.agency/review/).