Search for a command to run...
No comments yet. Be the first to comment.
If your worker is dropping jobs intermittently, sometimes crashing at boot with 'Error 111 connecting to redis:6379. Connection refused' and sometimes producing an empty result.json with no error at a
The finance lead asked why AWS charged us $2,100 for NAT gateway data processing last month. Our normal was around $400. Nothing in the release calendar explained it: no new services, no traffic bump
The AWS bill for EKS Compute went from $4,200 to $9,800 in five days. Same cluster, no launch, no traffic event, no team asking for headroom. kubectl get nodes returned 60 m5.4xlarge instances where t
The log line was 'database schema version 23 is dirty, refusing to start' and the pod exited immediately after printing it. The team had already tried a Helm rollback to the previous chart version. Th
Infraforge Insights
20 posts
Recovery playbooks and operational lessons from real SaaS infrastructure engagements. We write about Terraform state recovery, Kubernetes release stabilization, GitOps drift, audit readiness, and cloud cost triage, each post is grounded in a specific failure mode and the sequence we ran to recover.
Audience: senior engineers, platform leads, and CTOs at seed-to-Series B SaaS companies.
If a Helm rollback of your service left a subset of pods in CrashLoopBackOff with empty database credentials while the rest keep serving, you are looking at a ConfigMap deletion race, not a bad rollback. The window opened during the upgrade that failed: a ConfigMap templated as a pre-upgrade hook with before-hook-creation is deleted and recreated on every upgrade, and any pod that restarted inside that gap booted with empty envFrom values and cached a broken connection pool. The rollback is when you noticed, not what caused it, and it does not clean up after itself. The fix is not kubectl rollout restart deployment. That will nuke the healthy pods too. You verify the current ConfigMap is correct, kill only the pods whose env is empty, then patch the chart so the race cannot happen again.
Problem signals:
Why envFrom pods boot empty when the ConfigMap looks correct now
envFrom and env.valueFrom.configMapKeyRef resolve exactly once, at pod start. What happens when the ConfigMap is missing at that instant turns on one field: optional. With the default, optional: false, the kubelet refuses to start the container at all; the pod sits in CreateContainerConfigError with an Error: configmap not found event, which is loud and easy to diagnose. With optional: true, which plenty of charts set so that a missing config does not block a boot, the variable is simply absent, the client library reads it as an empty string, and the container starts and reports Running. That second case is the one this guide is about, and it is the only one that produces the split-brain symptom below. Nothing propagates later in either case. This is the part that confuses on-call: you look at the ConfigMap now, it is correct, so how can pods be running with empty values? Because those pods started at t=0 when the ConfigMap did not exist, and Kubernetes has no reconciliation loop that re-injects env into a running container.
The failed upgrade creates the window, not the rollback. If your ConfigMap carries a helm.sh/hook annotation (pre-install, pre-upgrade are the common ones) with a hook-delete-policy of before-hook-creation, Helm treats it as an ephemeral hook resource and deletes the existing copy before creating the new one on every upgrade. That delete-then-create is the gap. Helm fires hooks per lifecycle event, so a resource annotated pre-install,pre-upgrade is never touched by helm rollback, which runs pre-rollback and post-rollback only. Two consequences worth holding onto: the rollback did not open the window, and it does not close it either, so after rolling back to 47 the cluster is still holding revision 48's hook ConfigMap. On a healthy cluster with fast API server response, that window is 200 to 900 milliseconds. On a loaded one we have measured it at 4 seconds. Any pod that restarts inside it (crashloop backoff timer firing, HPA scale-up, node eviction) reads no ConfigMap and boots blank.
The 200ms to 4s window, opened by the upgrade's pre-upgrade hook, where a restart lands on a missing ConfigMap.
The tell that you are in this state and not something else: pods with identical spec, identical image, identical ConfigMap reference, have different runtime env. That does not happen from any cause other than the ConfigMap being absent at one pod's start time.
Check the chart state before you touch a single pod
Do not restart anything yet. First confirm the ConfigMap now holds the correct value, otherwise you are about to restart pods into the same broken state. Run the three checks in order.
# 1. What revision are we on and did it complete?
$ helm history payments-api -n prod
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
47 2024-11-03 14:22:11 UTC superseded payments-api-3.4.1 3.4.1 Upgrade complete
48 2024-11-03 14:41:07 UTC failed payments-api-3.5.0 3.5.0 Upgrade "payments-api" failed
49 2024-11-03 14:43:52 UTC deployed payments-api-3.4.1 3.4.1 Rollback to 47
# 2. Is the current ConfigMap the correct one?
$ kubectl get configmap payments-api-config -n prod -o jsonpath='{.data.DATABASE_URL}'
postgres://app:REDACTED@pg-primary.prod.svc:5432/payments?sslmode=require
# 3. Does its owner-reference or annotations still mark it as a hook?
$ kubectl get configmap payments-api-config -n prod -o yaml | grep -A2 'helm.sh/hook'
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-delete-policy: before-hook-creation
If line 3 returns a hook annotation, the chart bug that caused this is still present and will fire again on the next upgrade. Note the annotation lists no rollback event, which is why revision 49 left this object exactly as revision 48 created it.
If the DSN value is correct, you can proceed to selective pod recovery. If it is wrong or empty, you are looking at the failed release's config, which the rollback never replaced because the object is a hook, and you need to fix the ConfigMap directly with kubectl apply before doing anything else. We keep the last-known-good ConfigMap under source control precisely for this five-minute window.
Restart only the pods that booted into the gap
kubectl rollout restart deployment/payments-api will restart every pod including the ones currently serving traffic correctly. If you are already down to partial capacity and Postgres has connection limits, doubling the churn is how you turn a partial outage into a full one. Identify the broken pods from their actual runtime env, or from the failure the process logged when it read that env, and delete only those.
# The pod spec cannot answer this. A variable sourced from a ConfigMap keeps
# its valueFrom stanza and never has a resolved .value written back into it,
# and envFrom keys never appear in the spec at all. Read the runtime env, or
# read what the process logged when it read the runtime env.
# Running-but-wrong pods: read the process environment directly.
$ kubectl exec -n prod payments-api-7d9c8f4b6-4kx2m -- printenv DATABASE_URL
# (prints nothing and exits 1: the variable is absent)
# CrashLoopBackOff pods cannot be exec'd into. The previous container's log
# carries the same signal, and this is the sweep that found ours.
$ for pod in $(kubectl get pods -n prod -l app=payments-api -o name); do
if kubectl logs -n prod $pod --previous --tail=50 2>/dev/null | grep -q 'password authentication failed'; then
echo "BROKEN: $pod"
fi
done
BROKEN: pod/payments-api-7d9c8f4b6-4kx2m
BROKEN: pod/payments-api-7d9c8f4b6-9jvpd
BROKEN: pod/payments-api-7d9c8f4b6-nq7wr
# Delete only those three. The Deployment will recreate them and they will
# read the current (correct) ConfigMap on start.
$ kubectl delete pod -n prod payments-api-7d9c8f4b6-4kx2m payments-api-7d9c8f4b6-9jvpd payments-api-7d9c8f4b6-nq7wr
pod "payments-api-7d9c8f4b6-4kx2m" deleted
pod "payments-api-7d9c8f4b6-9jvpd" deleted
pod "payments-api-7d9c8f4b6-nq7wr" deleted
Runtime env, or the log line the process wrote when it read that env. The pod spec is not ground truth here.
The trap worth naming, because it is the first thing most people reach for: a jsonpath filter over .spec.containers[0].env[?(@.name=="DATABASE_URL")].value looks like it should work and cannot. Kubernetes resolves ConfigMap-sourced variables in the kubelet at container start and never writes the result back onto the PodSpec, so that filter returns empty for every ConfigMap-sourced pod, healthy or broken. On a 12-replica Deployment it flags all 12. kubectl debug with an ephemeral container reading /proc/1/environ is the third way in, useful when the container image has no shell for exec.
The chart change that closes the window permanently
The root cause was that the ConfigMap was templated as a hook. There is almost never a reason to do this for a ConfigMap that carries runtime configuration. Hooks are for one-shot resources: pre-upgrade DB migrations, post-install seed jobs. A ConfigMap that pods depend on at start time should be a regular chart resource with a stable lifecycle across upgrades and rollbacks.
# templates/configmap.yaml
# BEFORE: this is the bug
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "payments-api.fullname" . }}-config
annotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-delete-policy: before-hook-creation
data:
DATABASE_URL: {{ .Values.database.url | quote }}
# AFTER: regular resource, no hook annotations
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "payments-api.fullname" . }}-config
data:
DATABASE_URL: {{ .Values.database.url | quote }}
# templates/deployment.yaml: force a rolling restart when the ConfigMap changes
spec:
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
Remove the hook annotations. Add a checksum annotation to the Deployment pod template so config changes always trigger a controlled rolling restart.
The checksum annotation is the piece most charts get wrong. Without it, changing values in the ConfigMap does not restart pods at all, because Kubernetes sees no change to the Deployment spec. Teams then work around this with a manual kubectl rollout restart after every values change, or a dummy annotation bumped by hand on each deploy, both of which are the same churn the checksum gives you for free and both of which get forgotten under pressure. With the checksum, any ConfigMap content change updates the pod template hash, which triggers a normal Deployment rollout that respects maxSurge and maxUnavailable. There is a tradeoff: you get more rollouts than before, one per config change, and if your ConfigMap has values that churn (feature flags, dynamic settings), you should move those out into a separate ConfigMap that pods read at runtime rather than at start.
We have written the broader pattern for boot-time versus runtime configuration in our Kubernetes stabilization notes, and if this is landing during or right after a migration, the same race appears with any credential source that gets templated as a hook, including some migration recovery playbooks we have run this year.
Questions on-call keeps asking after this fires
Can I use kubectl rollout restart safely if the ConfigMap is now correct?
Yes, but only if you have the capacity to churn every pod and your database can absorb the reconnect storm. On a service with 12 replicas and a Postgres max_connections of 200 with pgbouncer in transaction mode, we have done full rollout restarts without incident. On a service that runs hot on connections, do the surgical delete instead.
Why does helm upgrade --atomic not prevent this?
--atomic rolls the release back when the upgrade fails, but the window has already opened by then. The pre-upgrade hook deletes and recreates the ConfigMap before any workload change is applied, so a pod can boot blank while the upgrade is still in flight. The rollback --atomic triggers then runs pre-rollback hooks only, which this ConfigMap is not annotated for, so it leaves the object untouched. --atomic is not a fix for this; removing the hook annotation is.
Does this affect Secret-sourced env the same way?
Yes, identically. envFrom on a Secret has the same start-time-only resolution. If you template Secrets as Helm hooks (some charts do, to inject generated passwords) you get the same race. Same fix: regular resource plus checksum annotation.
How do I know if the race actually fired versus something else broke my pods?
Two-pod comparison. Take one healthy pod and one crashing pod from the same ReplicaSet. If their runtime env for the same key differs (one has DATABASE_URL populated, the other has it empty), it is the ConfigMap race. If both have the same env and one still crashes, look elsewhere: image drift, node-local state, or a downstream dependency.
Can I use a mutating webhook or Reloader to auto-restart on ConfigMap changes instead of the checksum annotation?
Stakater Reloader works and we run it in several client clusters. The checksum annotation is simpler because it is part of the chart and needs no extra controller. If you already run Reloader for other reasons, use it. If you do not, do not install a controller to solve a one-line templating fix.
If you are staring at CrashLoopBackOff right now
The hard part of this recovery is not the kubectl commands. It is deciding which pods to touch when your dashboards are red and the pressure is to do the biggest hammer available. A rollout restart in the middle of a partial outage on a service under connection pressure is how the two-hour incident becomes the six-hour incident. The judgment call is: verify config, inspect env pod-by-pod, delete only what is actually broken, and only then fix the chart so this cannot recur.
At InfraForge we do this recovery with the on-call team on the bridge, in one working session, and we leave the chart patch and the checksum annotation merged before we sign off. If you would rather not run the surgical restart on production by yourself, book a 60-minute infrastructure review and we will walk it with your team today or tomorrow, chart fix included.
Originally published at https://infraforge.agency/insights/helm-rollback-configmap-race-safe-recovery/.
If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.