Problem signals:

• Savings Plan coverage percent drops sharply while total instance-hours stay inside normal variance
• A brand new instance-family line item appears in Cost Explorer that was at zero the prior month
• kubectl get nodes shows a family nobody added to any NodePool by name
• Cost Anomaly Detection has no notion of Savings Plan coverage; it models spend, so the stranded commitment is structurally invisible to it and the new on-demand family only alerts if it clears your subscription threshold
• Node count, pod count and latency are all flat across the window where the bill moved

What we checked first when Savings Plan coverage fell 59 points

Three explanations that all failed the arithmetic

The FinOps analyst at this healthtech SaaS (about 230 engineers, single us-east-1 footprint, four EKS clusters, EC2 running around $58k of a $92k monthly bill) pulls a coverage report on the first Monday of the month. June came back at 100%, consistent with the prior eight months. The partial-July pull came back at about 41%. Total instance-hours had moved from roughly 1,850 to 1,880 per day, which is inside the noise of ordinary customer onboarding. Only the coverage number had broken.

The platform lead's first answer was that nothing had changed, and it was an honest answer. Karpenter had gone from 1.0.6 to 1.1.4 in the June 30 maintenance window, two engineers had reviewed that PR, and the diff was a version string in a Terraform manifest. No NodePool CR had been edited. No large service had shipped. So we went looking for a billing-side explanation, and we burned most of a day on three of them.

The first was untagged usage falling out of the coverage denominator. That one dies on the API surface before you run a command: get-savings-plans-coverage exposes no tag dimension at all, so coverage data is never tag-scoped and tagging cannot move the number in either direction. Wrong tree. The second was a new linked account joining the Organization without Savings Plan sharing turned on, which genuinely does dilute aggregate coverage. aws organizations list-accounts returned the same eight accounts as June. Checking the sharing setting is not something you can do from a member account, which is worth saying because we tried it first: aws savingsplans describe-savings-plans returns only the plans the calling account owns, so a linked account never sees the payer's commitments through it, and the call says nothing about discount sharing either way. Sharing is a payer-level billing preference. We read it in the management account under Billing preferences, "Reserved Instance and Savings Plans discount sharing", and confirmed the discount was actually landing in the linked accounts from the payer, one call per account, filtering rather than grouping: aws ce get-savings-plans-coverage --time-period Start=2026-07-01,End=2026-07-22 --filter '{"Dimensions":{"Key":"LINKED_ACCOUNT","Values":["<acct-id>"]}}' --granularity MONTHLY. LINKED_ACCOUNT is a filter dimension on that API, not a GroupBy attribute; GroupBy accepts only INSTANCE_FAMILY, REGION or SERVICE, and --granularity is legal precisely because there is no GroupBy on the call. It was on. The third explanation was a service migrating from EC2-backed nodes onto Fargate, which an EC2 Instance-family plan does not reach. Both Fargate profiles were the same two that had existed for a year, and neither had picked up new work.

We also confirmed the boring one, that a plan had quietly retired. The account holds three EC2 Instance Savings Plans, one each for m5, m6i and r5, and all three had end dates in 2027 and 2028. Four explanations, four dead ends, and half a day gone against a billing cycle that closed on August 1.

The per-family coverage breakdown that ended the search in one query

22,250 hours on a family with zero coverage

The report the analyst pulls monthly is an aggregate. Aggregates hide exactly this class of problem, because a fleet swapping one family for another keeps every total roughly constant. Mid-afternoon we re-ran the coverage query grouped by instance family instead. That is not one extra argument on the same call, which cost us twenty minutes: get-savings-plans-coverage rejects --granularity when --group-by is set, so you issue one call per window and diff them yourself. It is also a spend API rather than a usage one, so the hours had to come from a second call to get-cost-and-usage.

# get-savings-plans-coverage rejects --granularity when --group-by is set,
# so run one call per window and diff them yourself.
aws ce get-savings-plans-coverage \
  --time-period Start=2026-06-01,End=2026-07-01 \
  --group-by Type=DIMENSION,Key=INSTANCE_FAMILY --output json

aws ce get-savings-plans-coverage \
  --time-period Start=2026-07-01,End=2026-07-22 \
  --group-by Type=DIMENSION,Key=INSTANCE_FAMILY --output json

# Coverage is spend-based. Each group returns only
# {CoveragePercentage, OnDemandCost, SpendCoveredBySavingsPlans, TotalCost}:
#   m5   "CoveragePercentage": "100" -> "100"
#   m7a  no usage in June            -> "0"   (~$10.3k OnDemandCost)
#   m6i  "CoveragePercentage": "100" -> "100"
#   r5   "CoveragePercentage": "100" -> "100"

# Hours are a different API, and get-cost-and-usage cannot group by
# INSTANCE_TYPE_FAMILY at all -- there it is a filter dimension only.
# Group by INSTANCE_TYPE and roll the sizes up into families yourself,
# or issue one call per family with
#   --filter '{"Dimensions":{"Key":"INSTANCE_TYPE_FAMILY","Values":["m7a"]}}'
aws ce get-cost-and-usage \
  --time-period Start=2026-06-01,End=2026-07-22 \
  --granularity MONTHLY --metrics UsageQuantity \
  --group-by Type=DIMENSION,Key=INSTANCE_TYPE --output json

# Sizes rolled up into families:
#          June, 30 days        Jul 1-22, 21 days
#   m5     47,000 (1,567/day)   11,280   (537/day)
#   m7a         0     (0/day)   22,250 (1,060/day)
#   m6i     2,300    (77/day)    1,610    (77/day)
#   r5      6,200   (207/day)    4,340   (207/day)
#   total  55,500 (1,850/day)   39,480 (1,880/day)

The aggregate said the fleet was fine. Grouped by family, 22,250 hours had walked onto a family with zero coverage while total hours per day did not move.

The July bucket is 21 days against June's 30, so read the per-day column and not the totals. Per day, m6i and r5 did not move at all, and the fleet total went from 1,850 to 1,880 hours a day, which is the variance the analyst had already dismissed. m5 had settled at roughly 29% of its June run rate; the 21-day average still reads higher than that because the swap itself sits inside the first day or two of the window. And a family that did not appear in June at all was carrying 22,250 hours at zero coverage. Confirming it in the cluster took one command, and the count told us how far it had gone.

kubectl get nodes \
  -o jsonpath='{range .items[*]}{.metadata.labels.node\.kubernetes\.io/instance-type}{"\n"}{end}' \
  | sort | uniq -c | sort -rn

     41 m7a.2xlarge
      7 m5.2xlarge

Nobody had typed m7a into a manifest anywhere in the repo. It was on 85% of the general-purpose pool.

The NodePool that governed that pool asked for instance-category In ["m", "r"], a set of allowed CPU counts, and an instance-family NotIn ["m5a", "m6a"] exclusion added in early 2024 after a numerical-precision problem with an AMD-based workload that had since been decommissioned. There was no generation constraint and no positive allow-list. The selector said, in effect, any m or r family AWS offers us, minus these two names. m7a sat inside that selector and no rule in the cluster objected, and it had been permitted long before the June 30 upgrade; nothing in that upgrade widened it. Karpenter's AWS provider enumerates the families it may pick from at runtime, from ec2:DescribeInstanceTypes and ec2:DescribeInstanceTypeOfferings against this account's own subnets and zones, then prices them from the AWS Pricing API. A family becomes selectable the day EC2 offers it in your zones: no controller release, no restart, no manifest change. The exclusion list was not out of date either, which is the uncomfortable part. m7a and m7i had both been generally available in us-east-1 since August 2023, five months before that NotIn was written. It was incomplete on the day it shipped, against families that already existed, because it named the two AMD generations the author had in mind and left everything else permitted.

Karpenter then did precisely its job, though not for the reason we assumed on day one. Our first theory was that it had found m7a cheaper. It had not, and the rate card says so plainly: in us-east-1, m7a.2xlarge lists at about \(0.4637/hr against \)0.384/hr for both m5.2xlarge and m6i.2xlarge, for the identical 8 vCPU / 32 GiB shape. Karpenter ranks candidate types on hourly price and, with consolidationPolicy: WhenEmptyOrUnderutilized, only replaces an underutilized node when the replacement is cheaper. A price-ranked replacement can never move a fleet from m5 onto on-demand m7a, and these hours billed at full on-demand list, so spot pricing was not in play either. What moved the fleet was capacity, and what asked for that much capacity at once was drift. The controller version bump was coincident rather than causal: bumping the chart restarts the controller pods and leaves existing NodeClaims alone, and Karpenter guards against mass drift across releases with the karpenter.sh/nodepool-hash-version annotation precisely so an upgrade does not roll the fleet. What did roll it went into the same maintenance window: the EC2NodeClass used an amiSelectorTerms alias rather than a pinned AMI ID, the alias resolved to a newly released AMI, and every node whose resolved AMI no longer matched was marked drifted and replaced. That needs no NodePool CR edit, which is why the version-string diff two engineers reviewed showed nothing. Karpenter hands EC2 Fleet a list of acceptable types in price order and Fleet launches what is actually available in the zone. m7a was on that list the whole time, ranked behind m5 and m6i on price; drift churned nodes fleet-wide, every replacement re-ran instance selection, and with that many launches compressed into one window the m5 and m6i capacity the fleet asked for was not there, so the requests fell through to the family that was. The hour table says how fast it finished. Steady-state m5 plus m7a is 1,596 hr/day once you subtract the flat 284 for m6i and r5, so with m5 settled at 29% of its June rate (454 hr/day), steady-state m7a is about 1,142 hr/day. The measured m7a average is 1,060 hr/day, 93% of steady state across the entire 21-day window, which only happens if the swap completed inside the first day or two of July. A migration trickling across three weeks would have landed near 12,000 hours, not 22,250. The m5 side agrees: 11,280 hours is only about a day and a half of full-rate m5 above the settled rate. What the provisioner cannot see either way is the Savings Plan portfolio, because commitments live in the billing system and are not part of the pricing signal it reads. Cheaper on the hourly rate card and cheaper after commitment amortization are two different questions, and Karpenter only answers the first.

Both halves of the loss run through a fleet that looks completely healthy from the cluster side.

That double charge is the part people miss. The commitment does not stop when you stop using it. \(18.40/hr kept flowing whether or not m5 nodes existed, and with m5 usage down to 29%, roughly \)310 a day of it bought nothing. Meanwhile the m7a hours billed at full on-demand list. Neither half appears as a line item labeled waste. The first shows up only as a coverage percent under 100, and the second shows up as a new usage type that nobody was watching for.

How we chose the fix with 9 days left on the billing cycle

Constrain the pool, buy a second commitment, or take the haircut

By late afternoon on day one we had three real options on the table and a cycle closing August 1. The analyst was on PTO from the 25th and the platform lead lost the 24th to a customer-facing latency incident, so the working window was about five days, not nine.

The change itself was small, and it was deliberately not another round of exclusions. A NotIn list plus a generation ceiling still permits every m and r family at generation 6 or below that nobody thought to name: m4, m5n, m5d, m5dn, m5zn, m6id, m6in, m6idn, r4, r5a, r5b, r5d, r5n, r6a, r6i, r6id, r6in. Not one of those is covered by any of the three plans, and the next thin-capacity hour lands the fleet on one of them exactly as it landed on m7a. m6a is itself a generation-6 family, which is why it still has to be named by hand, and that is the proof the generation bound cannot carry this on its own. So we replaced the deny-list with a positive allow-list naming exactly the three covered families, and kept the generation bound behind it as a second guard.

requirements:
  - key: karpenter.k8s.aws/instance-category
    operator: In
    values: ["m", "r"]
  - key: karpenter.k8s.aws/instance-family      # the load-bearing line
    operator: In
    values: ["m5", "m6i", "r5"]
  - key: karpenter.k8s.aws/instance-generation  # second guard, not the fix
    operator: Lt
    values: ["7"]

A NotIn list is a list of yesterday's problems, and a generation bound still admits every uncovered family under the ceiling. The In list is the one that holds against families nobody has named yet.

We planned it, had a second engineer read the plan, and applied to the largest cluster at 15:40 on day one. Karpenter picked up the new constraints on its next reconcile, well under a minute. Then the second-order problem showed up: unwinding this is itself a fleet-wide node replacement, and it moves no faster than the disruption budget and the drain times allow. The pool had disruption.budgets set to nodes: "10%", which is a setting we would defend on any production cluster, but read it correctly before you plan a window around it. That field is a concurrency cap, not a rate: at most 10% of the pool may be undergoing voluntary disruption at any one time, roughly four or five nodes here. There is no hourly-rate form of it; if you want a genuine time-based limit, that is a budgets entry with schedule and duration. Elapsed time is therefore set by how long each node takes to drain, PDB-gated pod eviction plus the replacement node registering, not by a per-hour quota. Draining 41 m7a nodes back onto m5 and m6i took a little over two hours. PDBs held, nothing paged, and every one of those hours billed m7a at on-demand while the m5 commitment was still under-consumed. We could have raised the budget to move faster. We did not, because trading a stable rollback for a couple of hundred dollars is a bad trade in a system where the failure mode we are recovering from is invisible to health checks.

The other thing that did not snap back was the number we were watching. Day two we finished the remaining three clusters by end of business. Day three, the trailing-window coverage query read 91.4%, not 100%, and that was correct rather than alarming. The coverage report looks back over a rolling window that still contained the uncovered m7a hours from earlier in July. Anyone treating a 24-hour coverage spot check as a pass/fail gate will either panic or declare victory early. Read the trend across several days, not the number on the day you shipped the fix.

FinOps closed the cycle at about $22,500 of overage across the 21 days before detection, plus roughly $2,400 during the mitigation week. Just under $25,000 on a month where the compute bill should have looked exactly like June's.

The 4 controls we shipped so the next new family cannot do this

Deny-lists inherit whatever the vendor ships next

The durable lesson is not about Karpenter and it is not about AMD. It is that a selector written as an exclusion list silently accepts every family you did not think to name, including the ones that already existed on the day you wrote it. Our NodePools were written as deny-lists, and a deny-list is a bet that you enumerated a set you do not control correctly, and that you will keep re-enumerating it every time AWS extends it. That bet loses on a long enough timeline, and the loss shows up in the billing system weeks later rather than in a failing test.

Four things landed in the platform IaC repo and the on-call docs over the two weeks after. The first is a rule in CI requiring instance-family to use operator: In with an enumerated list instead of NotIn. That flips the failure mode from silently adopting new families to silently ignoring them until a human adds one, which is the direction you want a cost-bearing default to fail in. We shipped it soft-failing for two weeks so teams could migrate, then made it blocking. It touched 11 NodePools across the four clusters. The second is a companion OPA policy, about 30 lines of rego, that fails the plan on any NodePool without an explicit karpenter.k8s.aws/instance-generation constraint. It is a backstop rather than the control that closes this hole, since a generation ceiling on its own still admits uncovered families below it, but the discovery set is now stated by us rather than inherited.

The third is the detection gap, and it is the one we would fix first if we could only do one. AWS Cost Anomaly Detection was watching the raw EC2 compute line at a \(500/day threshold and never fired. The explanation we reached for first, that the m5 commitment kept billing and offset the new on-demand spend, is backwards and worth correcting out loud: a Savings Plan charges its fixed hourly amount whether or not you consume it, so a constant charge cannot offset anything. The raw line did move. Of the roughly \)22,500 of overage, about $6,500 was commitment buying nothing at $310 a day, and the other \(16,000 or so was genuinely new on-demand spend, about \)760 a day, comfortably above the threshold, and it arrived as a step change in the first days of July rather than as a slow drift. That monitor should have caught it and did not, and we never got an account of its baselining out of it that we were willing to trust. What we could establish is what it does not evaluate at any threshold: coverage percent. So we stopped trying to tune a spend monitor into a coverage monitor.

So we wrote a nightly Lambda, about 80 lines of Python on an EventBridge schedule, that pulls the trailing seven days of coverage grouped by instance family and posts to the FinOps Slack channel on three conditions: a family carrying material uncovered spend that had none the week before, aggregate coverage moving more than 5 percentage points week over week, and Savings Plan utilization falling below 100%. Per-family coverage percentage is deliberately not one of them, and we learned that by writing the naive version first. Through this entire incident m5, m6i and r5 all read 100% and m7a read 0% from its first hour to its last, so a week-over-week delta on per-family coverage percent would never have crossed any threshold. What moved was which families carried the volume, and the aggregate, which went 100 to 41. Against this incident the alert as shipped would have fired within a day or two of the migration instead of 22, because m7a went from no usage at all to a full day of it. The fourth is a one-page runbook for the phrase "SP coverage dropped": run the coverage query grouped by family once per window and diff the two, pull the matching hours from get-cost-and-usage grouped by INSTANCE_TYPE and rolled up into families yourself (that API takes INSTANCE_TYPE_FAMILY only as a filter, never as a GroupBy), and if any family went from near zero to non-zero, go read the NodePool selector before you go read anything else.

We now treat any Kubernetes autoscaler as a component with a billing blast radius, not just a scheduling one, and we review its selectors during cost reviews rather than only during platform reviews. That framing is the same one we bring to Kubernetes and CI/CD stabilization work generally: the controller doing exactly what it was told is the most expensive kind of correct. If the shape of this feels familiar from the invoice side rather than the cluster side, the pattern is covered from that angle in cloud cost spikes.

Karpenter and Savings Plans: 4 questions worth answering before your next NodePool review

The questions the retro kept circling back to

These came up in the retro and have come up in every conversation we have had about this since.

• Does an EC2 Instance-family Savings Plan cover a different family in the same category? No. m5 and m7a are distinct families under the plan's scope rules, not size or generation variants of one another. An EC2 Instance Savings Plan commits to one instance family in one region, flexes only across size, OS and tenancy inside it, and buys a deeper discount in exchange for that narrowness. A Compute Savings Plan is the one that flexes across families, at a lower rate.
• Would Cost Anomaly Detection have caught this? Ours did not, at a \(500/day threshold, and not for the reason we first told ourselves. The commitment did not mask the delta: a Savings Plan bills the same fixed amount whether you consume it or not, so it cannot offset anything. The raw EC2 line really did rise, by about \)760/day of new on-demand spend, and it rose as a step change in the first days of July. It should have fired and it did not. Check what your monitor actually evaluates before you rely on it here; Cost Anomaly Detection models spend and has no notion of Savings Plan coverage, so a coverage collapse is structurally invisible to it, and the control that covers this failure is the per-family coverage alert.
• Is it safe to change a NodePool selector on a busy production cluster? It was for us, and the reason is disruption.budgets plus PDBs. Read the budget correctly first: nodes: "10%" caps how many nodes may be under voluntary disruption at once, not how many per hour. Karpenter replaced 41 nodes with four or five draining concurrently, a little over two hours end to end, with no service impact. Budget the elapsed time from your own drain and registration times, and resist raising the cap to finish faster; that is exactly where a cost incident turns into an availability incident.
• Should we just move to Compute Savings Plans? It removes this failure mode structurally, and it costs roughly 7 points of discount. Note that it is not a conversion: Savings Plans cannot be exchanged, modified or cancelled after the 7-day return window, so you either layer a Compute plan on top of current spend or replace the EC2 Instance plans as they expire. Nobody on the team was happy about the 7 points and nobody had a better answer. If your fleet is under an autoscaler that is free to pick families, price the immunity honestly rather than assuming the deeper discount is the cheaper option.

When the cluster looks healthy and the invoice does not agree

If your coverage number moved and your dashboards did not

The hard part of this class of incident is not the fix, which was a handful of lines in a NodePool. It is that every signal a platform team looks at daily stays green while the money leaves, and the one signal that would have caught it lives in a monthly FinOps artifact that the platform team does not own or read. Two functions each holding half the picture is how a 22-day detection lag happens to competent people.

We do this work on live clusters: reading autoscaler selectors against a commitment portfolio, finding the gap between what the scheduler optimizes and what the invoice charges, and leaving behind the CI policy and the alert so it does not come back the next time a new family lands in your zones.

If a coverage number just moved on you and the cluster looks fine, book an infrastructure review and we will get on a call the same day to find where your fleet went.

Originally published at https://infraforge.agency/insights/karpenter-savings-plan-coverage-drop/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• Karpenter's launched instance events run 5-10x baseline while disrupting via consolidation events sit at half of normal
• EC2 on-demand line items on the two largest instance sizes double while spot line items stay flat
• Node count climbs steadily through a quiet weekend with no matching increase in pod count or RPS
• kubectl finds karpenter.sh/do-not-disrupt: true on a majority of pods even though only a handful of workloads legitimately need it
• On-demand share of total node hours drifts from ~30% to >60% even though the NodePool weights spot at 100

The metric that redirected us away from HPAs

The launch rate was 7x normal; the consolidation rate was half

Our first read was wrong. The Monday FinOps digest fired at 08:15 UTC showing EC2 on-demand up 2.2x for the trailing 72 hours, and the platform lead's instinct was 'someone shipped an HPA that went sideways over the weekend.' We checked requests per second on the top 20 services; nothing moved more than 8% week over week. Total pod count was up about 40 pods against a fleet of 2,400. Not an HPA problem. Something was provisioning nodes without a matching demand signal, and the extra capacity was mostly on-demand even though the NodePool weighted spot at 100.

The evidence that pointed us at the right layer came from Karpenter's own controller logs. We asked it to summarise its own actions over the incident window:

kubectl -n karpenter logs deployment/karpenter --since=72h \
  | grep -E 'launched instance|disrupting via consolidation' \
  | awk '{print $6}' | sort | uniq -c

 2812 launched
  178 disrupting

# Baseline weekend, same window a fortnight prior:
  412 launched
  347 disrupting

Karpenter was doing both halves of its job. It was just doing one seven times too often and the other half as often as it should.

The provisioning path was healthy. There were no failed provisioning or unschedulable events. Karpenter was launching nodes about seven times as often as a normal weekend and consolidating them at half the usual rate, so capacity was arriving faster than it was leaving. Over 60 hours that gap accumulated into 98 extra nodes, most of them on-demand, sitting quietly and costing money.

One line in a shared chart, 34 tenants downstream

The chart shipped Friday at 15:04 UTC and Renovate did the rest

Once the question was 'why isn't consolidation firing', the answer came out fast. Karpenter's consolidation logic evaluates candidate nodes by checking whether every pod on the node can be safely rescheduled. Any node holding a pod with karpenter.sh/do-not-disrupt: true is immediately disqualified from consolidation until that pod is gone. The annotation exists for legitimate reasons: Kafka Streams state, long-running batches with expensive warmup, workloads that lose data on SIGTERM. In our cluster, four workloads owned it and were the only ones that should have had it.

The reality was different:

kubectl get pods --all-namespaces -o json \
  | jq -r '[.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true")] | length'

1450

kubectl get pods --all-namespaces -o json \
  | jq -r '[.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true") | .metadata.namespace] | unique | length'

38

1,450 pods across 38 namespaces carried the annotation. With pods distributed by the default scheduler, that pinned roughly 85% of nodes.

Our first suspicion was a mutating admission webhook. kubectl get mutatingwebhookconfigurations returned nothing new. The annotation was in the pod specs at the source. We picked one pod, walked back to its ReplicaSet, saw the annotation in the pod template, and ran git blame on the tenant's Deployment manifest. The blame line pointed at an internal chart bump: internal-charts/base-deployment moved from 2.7.4 to 2.7.5 on Friday at 19:47 UTC. The diff was three lines:

+ annotations:
+   karpenter.sh/do-not-disrupt: "true"

Commit message: 'add do-not-disrupt to prevent midday restarts during batch runs.'

The chart owner had one tenant whose Kafka Streams StatefulSet was losing ~90 seconds of state on every consolidation event. The narrow fix was to annotate that one StatefulSet, which required coordinating with the tenant. The wide fix was to put the annotation in the shared chart, which required coordinating with no one. They picked the wide fix. Our Renovate config had trust minor bumps from internal charts on auto-merge. Thirty-four tenant repos consumed base-deployment. Over the weekend, thirty-four PRs opened, thirty-four PRs merged, thirty-four ArgoCD syncs rolled deployments, and every rolled pod inherited the annotation.

A single opinionated chart plus a permissive auto-merge is the shape of the whole incident. Nothing in the loop was malicious. Every step was doing what it was configured to do.

The right recovery is on the pods, not the templates

The patch we almost ran would have rolled 1,450 pods

By 09:00 UTC Monday we had the diagnosis and a bad plan. The obvious move was to patch the Deployment template in each of the 34 tenants, removing the annotation at the source:

kubectl patch deployment <name> -n <ns> \
  -p '{"spec":{"template":{"metadata":{"annotations":{"karpenter.sh/do-not-disrupt":null}}}}}'

The command that looked surgical and was not.

We almost ran the loop. Then one of us asked the question that matters: does changing a template annotation trigger a rollout? It does. The Deployment controller's ComputeHash runs DeepHashObject over the entire PodTemplateSpec, and metadata.annotations is part of the template. Change any pod-template annotation and the template hash changes, which creates a new ReplicaSet, which rolls every pod. The cleanest proof of this is that kubectl rollout restart triggers a rollout precisely by stamping kubectl.kubernetes.io/restartedAt into spec.template.metadata.annotations. If template annotations were excluded from the hash, rollout restart could not work.

If we had run the patch loop across 34 tenants at 09:00 UTC on a Monday, we would have rolled about 1,450 pods in flight, cascaded PDB blocks into deploy pipelines, and (worst) triggered even more Karpenter provisioning as the rollouts churned. The recovery would have looked identical to a second, larger incident.

The correct move was to strip the annotation from the running pods, not from the template. A ReplicaSet reconciles pod count and pod ownership, not annotation drift on pods that already exist. So mutating a live pod's annotations is safe and does not trigger replacement. Karpenter re-checks consolidation eligibility on its next loop (roughly every 30 seconds) and starts disrupting the newly-eligible nodes on its own.

Here is what we ran, iterating one namespace at a time so we could abort if anything looked off:

for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  case "$ns" in kube-*|karpenter) continue ;; esac
  kubectl annotate pods -n "$ns" --all \
    karpenter.sh/do-not-disrupt- --overwrite 2>/dev/null || true
done

The trailing dash on the annotation key removes it. Total wall time across ~2,440 pods was 47 seconds.

Verification was two commands, one for the pod count and one for Karpenter's response:

kubectl get pods --all-namespaces -o json \
  | jq -r '[.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true")] | length'

24

kubectl -n karpenter logs deployment/karpenter --tail=200 \
  | grep 'disrupting via consolidation' | tail -5

...disrupting via consolidation, 3 candidates
...disrupting via consolidation, 5 candidates
...disrupting via consolidation, 2 candidates
...disrupting via consolidation, 4 candidates
...disrupting via consolidation, 3 candidates

Down to 24 annotated pods (the four legitimate workloads plus a 20-pod fraud-scoring batch six hours into its run) and Karpenter firing consolidation events within four minutes.

Over the next 90 minutes node count dropped from 198 to 141 and plateaued. The remaining 24 pods were pinning about 22 nodes: 20 fraud-batch pods spread across 18 nodes, and 4 legitimate pods on 4 more. Consolidation could not touch those. We coordinated a restart of the fraud batch with its owning team. It checkpointed cleanly and restarted onto 5 nodes, tight-packed by Karpenter's provisioning-time bin packing, which freed 13 nodes and took us to 128. Over the next 90 minutes natural pod churn shifted another 20 pods off nodes that had held only transient workloads, and Karpenter consolidated those too. Node count landed at 108 by 15:00 UTC, back inside baseline range.

Then we rolled out the fixed chart. base-deployment 2.7.6 removed the blanket annotation and gated it behind an explicit values.yaml opt-in (karpenter.doNotDisrupt: false default, true for the four legitimate tenants). This rollout DID replace pods, because it changed the template hash, and we scheduled it deliberately. Renovate opened PRs but no longer auto-merged them. The platform team merged them in batches of five with a ten-minute soak between batches so any interaction with PDBs or startup probes surfaced before the next wave. Six hours end to end, no SLO impact.

The changes that landed in the two weeks after

Three controls that would have caught this on Saturday

The postmortem produced three durable controls. Each one closes a specific step in the causal chain above.

The first is a Kyverno ClusterPolicy that only permits the annotation on pods that explicitly opt in via a label. It ran in Audit mode for two weeks, we watched the drift metric go to zero, then we flipped it to Enforce:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-karpenter-do-not-disrupt
spec:
  validationFailureAction: Enforce
  rules:
  - name: require-opt-in-label
    match:
      any:
      - resources:
          kinds: [Pod]
    preconditions:
      all:
      - key: "{{ request.object.metadata.annotations.\"karpenter.sh/do-not-disrupt\" || '' }}"
        operator: Equals
        value: "true"
    validate:
      message: "karpenter.sh/do-not-disrupt requires label karpenter.platform/opt-in=true"
      pattern:
        metadata:
          labels:
            karpenter.platform/opt-in: "true"

A future chart change that adds the annotation without the matching label is rejected at admission and never lands.

The second is an SLO alert on Karpenter's own consolidation metric. karpenter_disruption_actions_performed_total (the metric renamed from karpenter_deprovisioning_actions_performed_total around v0.32; the semantics are the same) was already in our Prometheus scrape config; nobody had put it on a dashboard. We added a Grafana panel showing consolidation actions per hour filtered to action="consolidation", and paged if the rate stayed below 20% of the 7-day rolling median for more than two hours during business hours. Backtesting against the incident, that alert would have fired around 04:00 UTC Saturday, roughly 8 hours in, instead of 60 hours in.

The third is a check in the shared chart's CI pipeline. Any minor bump that touches a scheduling-related annotation key (karpenter.sh/*, karpenter.k8s.aws/*, scheduler.alpha.kubernetes.io/*, node.kubernetes.io/*, cluster-autoscaler.kubernetes.io/*) sets a platform-review-required flag on the PR, and Renovate is configured to leave those PRs open rather than auto-merge them. Non-scheduling changes still flow through the previous auto-merge path unchanged. The tradeoff we now pay is about one platform-team review per quarter on this chart; the incident that check would have caught cost us roughly $3,500 in on-demand overage across the 60-hour window.

We considered tightening the NodePool's disruption.budgets and did not. The root cause was not consolidation being too eager; consolidation was fine when it was allowed to fire. Tightening budgets would slow legitimate consolidation without preventing another annotation-blast. We put a comment in the NodePool YAML pointing to the postmortem so a future engineer does not try to 'fix' the budgets in response to reading it.

FAQ: Karpenter consolidation and the do-not-disrupt annotation

Common questions we get about this pattern

Does karpenter.sh/do-not-disrupt pin just the pod or the whole node? It effectively pins the node. Consolidation eligibility is evaluated per-node, and any node holding at least one annotated pod is disqualified until that pod is gone. One annotated pod on a 32-vCPU node prevents the node from ever consolidating.

Would tightening PodDisruptionBudgets have prevented this? No. PDBs govern voluntary disruption of pods that Karpenter has already decided to touch. The annotation blocks node candidacy upstream of that check, so the eviction call PDBs guard never happens. PDBs sit downstream of the gate that closed here.

Can we detect this via CloudWatch or Cost Explorer alone? Not fast enough. Cost signals arrive at daily granularity at the earliest, and our FinOps digest ran weekly. The signal you want is Karpenter's own consolidation-actions rate from /metrics, which moves within minutes of the problem.

Does this pattern apply to Karpenter v1? The mechanism is the same. The annotation moved under different API groups across versions, and v1 uses karpenter.sh/do-not-disrupt on pods with identical semantics. The Deployment controller's PodTemplateSpec hash behavior is a Kubernetes property, not a Karpenter property, and it does not change.

What if we want every pod in a Deployment to be do-not-disrupt by design? Put the annotation in the template and accept that changing it triggers a rollout. Use maxSurge and PDBs to control that rollout the way you would for any other template change. The mistake in our incident was not that a template had the annotation; it was that 34 templates got it without the owning teams knowing.

Where Karpenter regressions get stuck

If your on-demand line jumped and no one shipped anything

The awkward thing about this class of Karpenter regression is that every piece looks correct in isolation. The annotation was a real feature intended for real workloads. The chart bump was a legitimate response to a real production pain. The Renovate auto-merge policy was tuned to reduce toil on well-behaved chart bumps. The NodePool was configured the way the docs recommend. It took the combination, plus a weekend, plus a weekly cost digest, to produce a doubled node count and a mid-four-figure overage. The diagnostic move that mattered was comparing Karpenter's launch rate against its consolidation rate. The recovery move that mattered was knowing which kubectl mutations trigger a rollout and which do not.

We have written more on this shape of failure in the Kubernetes and CI/CD stabilization pillar, and the specific pattern of a shared chart change cascading through GitOps sits in the ArgoCD and GitOps recovery cluster. If your Karpenter cluster is growing and you cannot see why, book an infrastructure review and we will pull the consolidation-actions metric together, walk your controller logs the same way we walked ours, and get you a diagnosis inside a single working day.

Originally published at https://infraforge.agency/insights/karpenter-consolidation-broken-by-do-not-disrupt-annotation/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• kubectl get nodes returns 'Unable to connect to the server: x509: certificate has expired or not yet valid'
• A CI or lifecycle script that used to work is now failing on every kubectl call, and the cluster has been up more than 11 months
• systemctl status k3s shows the service is active but the k3s.service journal has TLS handshake errors from the agent
• The kubeconfig at /etc/rancher/k3s/k3s.yaml decodes to a client cert whose notAfter is in the past
• Ephemeral or preview environments are stuck: nothing is being created, nothing is being deleted, and the queue is growing

What we walked into

119 namespaces that should have been gone by 6am

The cluster ran preview environments for a mid-size SaaS product. Every merged PR got a namespace, every closed PR got its namespace reaped by a cron that shelled out to kubectl. Simple, effective, and completely dependent on kubectl being able to talk to the API server. On the morning of the incident the reap job had been erroring every minute for about six hours before the platform on-call noticed the namespace count on the Grafana panel had gone flat instead of sawtoothing.

The error in the cron log was one line, repeated:

Unable to connect to the server: x509: certificate has expired or not yet valid: current time 2025-04-12T06:14:22Z is after 2025-04-11T18:02:17Z

The API server had been unreachable for twelve hours and twelve minutes by the time we opened the log.

This is a single-server k3s install with one agent node, running on two small EC2 instances. It had been up for 366 days, one day longer than its own certificates were valid for. Nobody had ever rotated a certificate on it. Nobody had written down that k3s issues its own internal client certs with a 12 month lifetime and quietly renews them on restart if they are within 90 days of expiry. Ours were nowhere near that window when the server last restarted, because the last restart was the kernel patch cycle that brought this cluster up in the first place, twelve months earlier. So they aged out on schedule and the cluster locked itself out.

The first thing we tried, and why it did not work

The kubeconfig regen that fixed nothing

The instinct on the first responder was correct-looking and wrong. They assumed the kubeconfig on the CI runner had drifted, copied a fresh /etc/rancher/k3s/k3s.yaml from the server, rewrote the server URL, and reran the reap. Same x509 error. They tried it a second time with KUBECONFIG pointed explicitly at the new file. Same error.

The reason it did not work is the piece that catches people out with k3s specifically. The k3s.yaml on disk is not a pointer to cert files; it is a self-contained kubeconfig with client-certificate-data and client-key-data base64-embedded directly in it. Regenerating the file by copying it from the server just copies the same expired bytes to a new path. The cert material lives at /var/lib/rancher/k3s/server/tls/client-admin.crt, and that is what had actually expired. Until that file was rotated, every kubeconfig on every machine, freshly copied or not, was carrying the same dead cert.

This is the point where we stopped guessing and asked openssl what it saw.

How we confirmed the actual expiry before touching anything

openssl said notAfter=Apr 11

Before running any rotation command we wanted to see the cert dates ourselves. Two commands, run on the k3s server:

$ sudo openssl x509 -noout -dates -in /var/lib/rancher/k3s/server/tls/client-admin.crt
notBefore=Apr 11 18:02:17 2024 GMT
notAfter=Apr 11 18:02:17 2025 GMT

$ for f in /var/lib/rancher/k3s/server/tls/client-*.crt /var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt; do
>   echo "$f"
>   sudo openssl x509 -noout -enddate -in "$f"
> done

The client-admin cert expired at 18:02 the previous evening. Every other client cert on the server had the same notAfter within a few seconds.

The serving cert for the API server itself was also expired. That mattered for the restart order later, because a k3s server with an expired serving cert will start, but agents cannot re-establish TLS to it until the rotation writes new material and the server picks it up. If we had restarted the agent first, or in parallel, we would have watched it fail to rejoin and then chased a second ghost.

Related reading: the same failure mode shows up during cluster migrations when a snapshot from an old server gets restored past the cert lifetime. We wrote about that pattern in the migration recovery notes.

The recovery sequence that actually worked

Rotate, restart server, restart agent, in that order

k3s ships a rotation subcommand that regenerates the internal certs on disk. It does not restart the service, and it does not touch k3s.yaml; the restart in step 3 is what rewrites that file. The second trap is everything downstream of the server, which we get to below. The full sequence, run as root on the server, was:

# 1. Stop the server so nothing is holding the old TLS material.
sudo systemctl stop k3s

# 2. Rotate all internal certs. On k3s v1.28+ this rewrites everything
# under /var/lib/rancher/k3s/server/tls including client-admin.crt.
sudo k3s certificate rotate

# 3. Bring the server back up. It will pick up the new certs on start.
sudo systemctl start k3s

# 4. Wait until the API is responsive with the local (root-owned) kubeconfig.
sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml k3s kubectl get nodes

# 5. On the agent node, restart so it re-handshakes with the new server cert.
sudo systemctl restart k3s-agent

Stop first, rotate, start, verify, then touch the agent. The k3s docs are explicit that the server has to be stopped before rotating, and we have separately seen a rotate against a live server leave the API server holding the old serving cert in memory until a restart anyway.

Step 4 worked from root on the server. On the CI runner, kubectl was still failing. This is where the k3s.yaml quirk from earlier comes back, with the sign flipped. k3s rewrites /etc/rancher/k3s/k3s.yaml on every start, so by the time step 4 ran, the server's own kubeconfig already carried the new embedded material; that is exactly why step 4 worked. Nothing rewrites the copies. Every kubeconfig we had ever scp'd to a runner or a laptop still held the dead bytes, and no amount of restarting the server was going to reach them. We had to export the fresh one and re-seed each consumer by hand:

# On the server, export the kubeconfig k3s rewrote when it started.
# config view --raw only prints what is already on disk; the restart
# in step 3 is what put the fresh cert material there.
sudo k3s kubectl config view --raw > /tmp/k3s-fresh.yaml

# Verify the embedded client cert is not the dead one.
grep client-certificate-data /tmp/k3s-fresh.yaml \
  | awk '{print $2}' | base64 -d \
  | openssl x509 -noout -dates
# notAfter=Apr 12 06:41:07 2026 GMT   <-- future date, good.

# Distribute to the CI runner's kubeconfig path. In our case:
scp /tmp/k3s-fresh.yaml ci-runner:/etc/lifecycle/kube/config
ssh ci-runner 'sudo chown lifecycle:lifecycle /etc/lifecycle/kube/config && sudo chmod 600 /etc/lifecycle/kube/config'

The server's own kubeconfig is correct after the restart. The copies are not, and nothing on the cluster knows they exist. Every consumer holding one has to be re-seeded by hand.

Elapsed time from opening the log to a successful kubectl get nodes on the CI runner was 47 minutes. Most of that was the first-fix detour. The rotation itself took about 90 seconds; the server restart took under 20.

The left branch is where we spent 30 wasted minutes. The right branch is what the runbook now says to do first.

FAQ

Questions the team asked when we posted the internal postmortem

Can I just add insecure-skip-tls-verify to the kubeconfig and move on?

No. The API server is presenting an expired serving cert, but it is also validating incoming client certs. Once the client-admin cert is past notAfter, the server rejects the request during TLS with the same x509 error even if the client stopped verifying. Skipping verification on the client side does not help you here. It also leaves the agent unable to reconnect independently.

Does k3s not auto-renew certs?

It does, but only if the server restarts while the certs are inside the 90 day renewal window. If the cluster runs 12 months straight with no restart, or the last restart happened more than 90 days before expiry, the renewal window is missed and the certs age out. That is exactly what happened to us.

Do I need to restart etcd or anything else on a single-server k3s?

On single-server k3s with the default embedded SQLite or embedded etcd, systemctl restart k3s covers everything the server needs. On an HA k3s with external etcd or multiple servers, you rotate and restart one server at a time and let the API stay reachable through the others. That is a different story and worth its own runbook.

Will the pods restart when I bounce the k3s server?

No. Workload pods run under containerd and are not managed by the k3s systemd unit's lifecycle in a way that restarts them. During our recovery, application pods kept serving traffic the entire time the API was down. Only the control plane was affected. Anything using in-cluster kubeconfigs (operators, controllers) will reconnect once the API comes back.

Is the agent's own node cert also rotated?

It needs its own attention. k3s certificate rotate only rewrites files under /var/lib/rancher/k3s/server/tls on the machine you run it on; it does not reach into the agent's /var/lib/rancher/k3s/agent/. Restarting k3s-agent works because the agent cannot authenticate with its expired client cert, falls back to re-bootstrapping with the node token, and the server issues it a fresh one. That is a fallback path, not a rotation. The deterministic version is to run k3s certificate rotate on the agent node too, servers first, then agents. Either way step 5 is not optional.

Where we help

If your k3s cluster is past its first birthday

The two things we changed after this incident were not glamorous. We added a nightly check that runs openssl x509 -noout -enddate against every cert under /var/lib/rancher/k3s/server/tls and every embedded client-certificate-data in every kubeconfig we own, and pages if any is under 30 days from expiry. It is a 42 line shell script and it has already caught the same class of problem on a second cluster that a client acquired through a merger. We also scheduled a monthly forced restart of the k3s server during a maintenance window. Quarterly is not enough: restarts 91 days apart land inside the last-90-days renewal window at most once per 12 month cert, and only by a day or two. Monthly gives three restarts inside the window. Neither of these prevents the failure by itself; together they close the door.

The hard part of this kind of work is not running k3s certificate rotate. It is knowing before you touch anything which cert is actually expired, what order the components have to come back in for this specific k3s version, and which files the rotation does and does not rewrite. We do this often enough on inherited k3s and k0s clusters, especially ones that came along with an acquisition or a platform-team handover, to know where the traps are on each version. If you have a cluster that is past its first birthday and nobody on the team can point to the last time certs were rotated, book an infrastructure review and we will spend a 60 minute call walking the cert state with you before it fails at 3am. If you are staring at x509: certificate has expired in a log right now, say so in the request and we will be on a bridge the same day.

Originally published at https://infraforge.agency/insights/k3s-certificate-expiration-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• terraform plan shows -/+ replace on resources actively serving production traffic
• The plan wants to destroy resources that were created out-of-band during a recent incident
• aws describe commands confirm the resources exist and are healthy, but terraform state does not know about them
• The team has stopped running terraform plan against this workspace because the output is too scary to act on

Five destroy-and-replace actions against live production

The plan wanted to destroy the primary database

The workspace was the one that owned the data layer for the primary application. Its state held the pre-failover Aurora cluster, the RDS proxy, and three security groups, all with us-east-1 ARNs. During the incident someone had updated the module's provider region from a variable to a hardcoded 'us-west-2', trying to bring the manually-created resources under management. That change alone did not import anything. What it did was make terraform want to replace every resource in the module, because state said us-east-1 and config now said us-west-2. Five resources in the module, five replacements, five destroys and five creates in the summary.

$ terraform plan

Terraform will perform the following actions:

  # module.data_layer.aws_rds_cluster.primary must be replaced
-/+ resource "aws_rds_cluster" "primary" {
      ~ arn                = "arn:aws:rds:us-east-1:...:cluster:app-primary" -> (known after apply)
      ~ endpoint           = "app-primary.cluster-abc.us-east-1.rds.amazonaws.com" -> (known after apply)
      ~ id                 = "app-primary" -> (known after apply)
      # (27 unchanged attributes hidden)
    }

  # module.data_layer.aws_db_proxy.primary must be replaced
-/+ resource "aws_db_proxy" "primary" {
      ~ arn      = "arn:aws:rds:us-east-1:...:db-proxy:prx-0a1b2c3d4e5f60718" -> (known after apply)
      ~ endpoint = "app-primary-proxy.proxy-abc.us-east-1.rds.amazonaws.com" -> (known after apply)
      # (14 unchanged attributes hidden)
    }

  # module.data_layer.aws_security_group.db_ingress[0] must be replaced
-/+ resource "aws_security_group" "db_ingress" {
      ~ arn = "arn:aws:ec2:us-east-1:...:security-group/sg-0abcdef1234567890" -> (known after apply)
      ~ id  = "sg-0abcdef1234567890" -> (known after apply)
    }

  # module.data_layer.aws_security_group.db_ingress[1] must be replaced
-/+ resource "aws_security_group" "db_ingress" { ... }

  # module.data_layer.aws_security_group.db_ingress[2] must be replaced
-/+ resource "aws_security_group" "db_ingress" { ... }

Plan: 5 to add, 0 to change, 5 to destroy.

The plan that stopped the room at 08:47

The apply would have done exactly what the plan said. It would have called DeleteDBCluster on the us-east-1 cluster, which was still up and still holding connections from the pools that had not been cut over yet. It would have called CreateDBCluster in us-west-2 with the identifier 'app-primary', which was already the identifier on the live cluster that SRE had created at 02:14. That second call would have returned DBClusterAlreadyExistsFault, and the state would have been mid-transaction, with the us-east-1 cluster gone and the us-west-2 cluster still not tracked. The same shape of failure was queued up for the proxy and the security groups, each with its own AWS-side name-collision error.

Why terraform state rm looked right and was wrong

What we almost did that would have made it worse

The reflex in the room was to run terraform state rm module.data_layer.aws_rds_cluster.primary on all 5 entries, then re-plan. State would forget the us-east-1 resources, plan would show 5 creates in us-west-2, and terraform would be back in a consistent-looking place. We had the commands typed. We did not run them, because someone asked the question that saved the afternoon: what does the create actually do when the resource already exists?

The answer is: it fails, and it fails deterministically. CreateDBCluster refuses an identifier that already exists in the region and returns DBClusterAlreadyExistsFault whether or not the properties you are asking for match the live resource. There is no adopt-if-identical path in the AWS API. CreateDBProxy behaves the same way, and duplicate security group names inside a VPC come back as InvalidGroup.Duplicate. Worse, a replace is a destroy followed by a create, so the destroy half lands first: the us-east-1 cluster is gone, the create fails, and state holds five addresses whose real resources terraform just deleted and could not recreate. Sorting that out is a bad afternoon and a worse changelog entry.

The right question is not 'how do we make terraform stop wanting to destroy things'. It is 'how do we get the resources that ARE running into state under terraform management, with the state entries reflecting reality'. Different question, different answer. state rm alone does not answer it, because the plan step after state rm still wants to create. Import blocks answer it, because they tell terraform 'this thing already exists, adopt it, do not create it'.

The Terraform 1.5 import block

Import blocks over state manipulation

Terraform 1.5 shipped import {} blocks in June 2023. Before that, the only tool was terraform import at the CLI, one resource at a time, with no config generation. You had to write the resource block by hand, run the import, then plan until it was empty. For 5 resources that is a slow afternoon. For 30, which is the size of the recovery we have done more often after a full-region failover, it is a week and a half.

The block form lives in HCL and terraform reads it during plan. Our module already declared all 5 resources (the only thing that had changed was the provider region), so there was nothing to generate: the plan matched each import block to the resource block already in the module and staged the adoption for the next apply. -generate-config-out is the companion flag for the other case, when you are adopting something you have no configuration for at all, and it writes a starter resource block for each import block that has no matching config.

# imports.tf (temporary, delete after one apply cycle)

import {
  to = module.data_layer.aws_rds_cluster.primary
  id = "app-primary"
}

import {
  to = module.data_layer.aws_db_proxy.primary
  id = "app-primary-proxy"
}

import {
  to = module.data_layer.aws_security_group.db_ingress[0]
  id = "sg-0a1b2c3d4e5f60718"
}

import {
  to = module.data_layer.aws_security_group.db_ingress[1]
  id = "sg-1a2b3c4d5e6f70819"
}

import {
  to = module.data_layer.aws_security_group.db_ingress[2]
  id = "sg-2b3c4d5e6f708192a"
}

One import block per resource; the id is the AWS resource ID, the to is the terraform module address

There is a state cleanup step first. The us-east-1 entries at those module addresses have to go, otherwise the import collides with an existing state entry and errors with 'resource already managed by Terraform'. We ran terraform state rm on the 5 stale entries. This is safe here because those state entries no longer match anything we intend to manage from this workspace (the us-east-1 resources are being handled by a separate decommission workspace). state rm is only dangerous when the plan step that follows tries to create things. The plan step that follows here only reads: import blocks are resolved during plan, and nothing touches AWS until apply.

$ terraform plan

Terraform will perform the following actions:

  # module.data_layer.aws_rds_cluster.primary will be imported
    resource "aws_rds_cluster" "primary" {
        cluster_identifier = "app-primary"
        engine             = "aurora-postgresql"
        engine_version     = "15.3"
        # ... 24 more attributes read from AWS
    }

  # module.data_layer.aws_db_proxy.primary will be imported
  # module.data_layer.aws_security_group.db_ingress[0] will be imported
  # module.data_layer.aws_security_group.db_ingress[1] will be imported
  # module.data_layer.aws_security_group.db_ingress[2] will be imported

Plan: 5 to import, 0 to add, 0 to change, 0 to destroy.

The successful plan after state rm + import blocks: 5 imports, 0 creates, 0 destroys

Read the summary line: 5 to import, 0 to add, 0 to change, 0 to destroy. That is the shape you want, and any add or destroy in it means an import block is pointing at the wrong address, or the module's attributes have drifted from what SRE actually built at 02:14. Fix that before you apply, not after. Commit imports.tf. Apply. The apply runs the 5 imports. After that, delete the import {} blocks. Once the address is in state they are no-ops on every subsequent plan, so leaving them in place is not an error, it is just noise that outlives the reason it was added.

How to align config to reality without overwriting the live resource

The generated HCL is 95% right and the last 5% matters

The generated config reflects what is running. That includes anything SRE added at 02:14 that was not in the pre-incident module. On this recovery, the next plan showed three drifts, all small, all load-bearing.

$ terraform plan

Terraform will perform the following actions:

  # module.data_layer.aws_rds_cluster.primary will be updated in-place
  ~ resource "aws_rds_cluster" "primary" {
      ~ backup_retention_period = 1 -> 7
      ~ tags = {
          - "incident-response" = "2024-11-14" -> null
          - "incident-owner"    = "sre-oncall"  -> null
        }
    }

  # module.data_layer.aws_db_proxy.primary will be updated in-place
  ~ resource "aws_db_proxy" "primary" {
      ~ debug_logging = true -> false
    }

Plan: 0 to add, 2 to change, 0 to destroy.

The next plan after import: the live state differs from the pre-incident module in three specific places

Two tags SRE had added at create time so the resources would be enumerable later. backup_retention_period at 1 day (the AWS default, because the incident timer was ticking and SRE clicked through the console fast) instead of the module's 7. debug_logging = true on the proxy, because SRE wanted verbose logs during the incident. Every one of these was a live state difference from what the module said the resource should look like.

For each drift item the decision is: keep the live state (add to config) or accept the plan (module wins)? On this recovery we kept the two tags for a followup and added them to the module's default tag map. We bumped backup_retention_period in the module to 7 to match the module's intent, accepting the plan on that one. We kept debug_logging=true for one more week and let the plan sit non-empty on that single field until the incident postmortem was done, then flipped it. The tempting move is to apply the whole plan at once and let the module reset everything. Do not do that without walking through each item. The point of the import was to bring the live resource under management, not to overwrite it with pre-incident config that no longer applies.

The other lesson from this recovery went into the incident runbook. When SRE creates resources by hand at 02:14, they now write the resource IDs into incident-resources.txt at the same time and commit it before the incident closes. That file becomes the source of truth for the import blocks the next morning, and cuts the recovery from 'grep CloudTrail for what got created between 02:00 and 03:00' to 'read the file, generate the imports'.

The 30-minute triage for a plan you cannot safely apply

If your terraform plan looks like this right now

The mechanical part of this recovery, the state rm and the import blocks, is a few hours of work. The hard part is the days before that, when the plan sits red and every engineer is afraid to touch it. Meanwhile someone else adds another out-of-band resource, and the next plan gets 12 lines longer, and the fear compounds. We have seen workspaces sit in this state for six months. The team stops running plan against them at all, which is its own kind of terrifying, because now nobody knows what actually differs from IaC.

We run these engagements every week. The failover-import case we have done three times this quarter alone, plus the 'someone tagged everything in the console during audit prep' variant, plus a half-dozen other shapes of out-of-band drift. Every one follows the same pattern: get the running resources into terraform without touching production, then decide what to reconcile and in what order. The reason it works is that the mechanical steps and the judgment steps are kept separate, so nobody applies the plan at 08:47 to make the red output go away.

If your workspace has a plan you cannot apply and no one is sure how it got there, book an infrastructure review and we will be on a bridge with you the same day, starting with a 30-minute diagnostic call to figure out whether the safe recovery is import blocks, moved blocks, a state file surgery, or a fresh workspace. For the class of problem this fits into, see the terraform apply fear pattern and the terraform state recovery playbook.

Originally published at https://infraforge.agency/insights/terraform-plan-destroy-live-failover-resources/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• kubectl get pods shows 3 of 12 replicas in CrashLoopBackOff with restart counts climbing (14, 17, 19) while the rest sit at 0 restarts
• Application logs on the crashing pods show pq: password authentication failed for user "" or dial tcp: missing address
• kubectl exec broken-pod -- printenv DATABASE_URL prints nothing and exits 1, while the same command on a healthy pod prints the DSN
• helm history shows a recent rollback from revision N to N-1 within the last hour
• kubectl get configmap -o yaml shows the correct DSN, so the state looks fine from the cluster's perspective and the alerting is confusing

The propagation window nobody documents

Why envFrom pods boot empty when the ConfigMap looks correct now

envFrom and env.valueFrom.configMapKeyRef resolve exactly once, at pod start. What happens when the ConfigMap is missing at that instant turns on one field: optional. With the default, optional: false, the kubelet refuses to start the container at all; the pod sits in CreateContainerConfigError with an Error: configmap not found event, which is loud and easy to diagnose. With optional: true, which plenty of charts set so that a missing config does not block a boot, the variable is simply absent, the client library reads it as an empty string, and the container starts and reports Running. That second case is the one this guide is about, and it is the only one that produces the split-brain symptom below. Nothing propagates later in either case. This is the part that confuses on-call: you look at the ConfigMap now, it is correct, so how can pods be running with empty values? Because those pods started at t=0 when the ConfigMap did not exist, and Kubernetes has no reconciliation loop that re-injects env into a running container.

The failed upgrade creates the window, not the rollback. If your ConfigMap carries a helm.sh/hook annotation (pre-install, pre-upgrade are the common ones) with a hook-delete-policy of before-hook-creation, Helm treats it as an ephemeral hook resource and deletes the existing copy before creating the new one on every upgrade. That delete-then-create is the gap. Helm fires hooks per lifecycle event, so a resource annotated pre-install,pre-upgrade is never touched by helm rollback, which runs pre-rollback and post-rollback only. Two consequences worth holding onto: the rollback did not open the window, and it does not close it either, so after rolling back to 47 the cluster is still holding revision 48's hook ConfigMap. On a healthy cluster with fast API server response, that window is 200 to 900 milliseconds. On a loaded one we have measured it at 4 seconds. Any pod that restarts inside it (crashloop backoff timer firing, HPA scale-up, node eviction) reads no ConfigMap and boots blank.

The 200ms to 4s window, opened by the upgrade's pre-upgrade hook, where a restart lands on a missing ConfigMap.

The tell that you are in this state and not something else: pods with identical spec, identical image, identical ConfigMap reference, have different runtime env. That does not happen from any cause other than the ConfigMap being absent at one pod's start time.

Confirm the rollback actually landed the correct config

Check the chart state before you touch a single pod

Do not restart anything yet. First confirm the ConfigMap now holds the correct value, otherwise you are about to restart pods into the same broken state. Run the three checks in order.

# 1. What revision are we on and did it complete?
$ helm history payments-api -n prod
REVISION  UPDATED                   STATUS      CHART              APP VERSION  DESCRIPTION
47        2024-11-03 14:22:11 UTC  superseded  payments-api-3.4.1 3.4.1        Upgrade complete
48        2024-11-03 14:41:07 UTC  failed      payments-api-3.5.0 3.5.0        Upgrade "payments-api" failed
49        2024-11-03 14:43:52 UTC  deployed    payments-api-3.4.1 3.4.1        Rollback to 47

# 2. Is the current ConfigMap the correct one?
$ kubectl get configmap payments-api-config -n prod -o jsonpath='{.data.DATABASE_URL}'
postgres://app:REDACTED@pg-primary.prod.svc:5432/payments?sslmode=require

# 3. Does its owner-reference or annotations still mark it as a hook?
$ kubectl get configmap payments-api-config -n prod -o yaml | grep -A2 'helm.sh/hook'
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation

If line 3 returns a hook annotation, the chart bug that caused this is still present and will fire again on the next upgrade. Note the annotation lists no rollback event, which is why revision 49 left this object exactly as revision 48 created it.

If the DSN value is correct, you can proceed to selective pod recovery. If it is wrong or empty, you are looking at the failed release's config, which the rollback never replaced because the object is a hook, and you need to fix the ConfigMap directly with kubectl apply before doing anything else. We keep the last-known-good ConfigMap under source control precisely for this five-minute window.

The surgical restart, not kubectl rollout restart

Restart only the pods that booted into the gap

kubectl rollout restart deployment/payments-api will restart every pod including the ones currently serving traffic correctly. If you are already down to partial capacity and Postgres has connection limits, doubling the churn is how you turn a partial outage into a full one. Identify the broken pods from their actual runtime env, or from the failure the process logged when it read that env, and delete only those.

# The pod spec cannot answer this. A variable sourced from a ConfigMap keeps
# its valueFrom stanza and never has a resolved .value written back into it,
# and envFrom keys never appear in the spec at all. Read the runtime env, or
# read what the process logged when it read the runtime env.

# Running-but-wrong pods: read the process environment directly.
$ kubectl exec -n prod payments-api-7d9c8f4b6-4kx2m -- printenv DATABASE_URL
# (prints nothing and exits 1: the variable is absent)

# CrashLoopBackOff pods cannot be exec'd into. The previous container's log
# carries the same signal, and this is the sweep that found ours.
$ for pod in $(kubectl get pods -n prod -l app=payments-api -o name); do
    if kubectl logs -n prod $pod --previous --tail=50 2>/dev/null | grep -q 'password authentication failed'; then
      echo "BROKEN: $pod"
    fi
  done
BROKEN: pod/payments-api-7d9c8f4b6-4kx2m
BROKEN: pod/payments-api-7d9c8f4b6-9jvpd
BROKEN: pod/payments-api-7d9c8f4b6-nq7wr

# Delete only those three. The Deployment will recreate them and they will
# read the current (correct) ConfigMap on start.
$ kubectl delete pod -n prod payments-api-7d9c8f4b6-4kx2m payments-api-7d9c8f4b6-9jvpd payments-api-7d9c8f4b6-nq7wr
pod "payments-api-7d9c8f4b6-4kx2m" deleted
pod "payments-api-7d9c8f4b6-9jvpd" deleted
pod "payments-api-7d9c8f4b6-nq7wr" deleted

Runtime env, or the log line the process wrote when it read that env. The pod spec is not ground truth here.

The trap worth naming, because it is the first thing most people reach for: a jsonpath filter over .spec.containers[0].env[?(@.name=="DATABASE_URL")].value looks like it should work and cannot. Kubernetes resolves ConfigMap-sourced variables in the kubelet at container start and never writes the result back onto the PodSpec, so that filter returns empty for every ConfigMap-sourced pod, healthy or broken. On a 12-replica Deployment it flags all 12. kubectl debug with an ephemeral container reading /proc/1/environ is the third way in, useful when the container image has no shell for exec.

• Delete the broken pods in batches of one or two, not all at once, to protect the connection pool of the pods that are still healthy.
• Watch the new pods reach Ready before deleting the next batch: kubectl get pods -n prod -l app=payments-api -w.
• If your Deployment has maxUnavailable set aggressively, note that manual pod deletes bypass rollout controls, so you set the pace by hand.

Stop treating the ConfigMap as a Helm hook

The chart change that closes the window permanently

The root cause was that the ConfigMap was templated as a hook. There is almost never a reason to do this for a ConfigMap that carries runtime configuration. Hooks are for one-shot resources: pre-upgrade DB migrations, post-install seed jobs. A ConfigMap that pods depend on at start time should be a regular chart resource with a stable lifecycle across upgrades and rollbacks.

# templates/configmap.yaml
# BEFORE: this is the bug
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "payments-api.fullname" . }}-config
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation
data:
  DATABASE_URL: {{ .Values.database.url | quote }}

# AFTER: regular resource, no hook annotations
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "payments-api.fullname" . }}-config
data:
  DATABASE_URL: {{ .Values.database.url | quote }}

# templates/deployment.yaml: force a rolling restart when the ConfigMap changes
spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

Remove the hook annotations. Add a checksum annotation to the Deployment pod template so config changes always trigger a controlled rolling restart.

The checksum annotation is the piece most charts get wrong. Without it, changing values in the ConfigMap does not restart pods at all, because Kubernetes sees no change to the Deployment spec. Teams then work around this with a manual kubectl rollout restart after every values change, or a dummy annotation bumped by hand on each deploy, both of which are the same churn the checksum gives you for free and both of which get forgotten under pressure. With the checksum, any ConfigMap content change updates the pod template hash, which triggers a normal Deployment rollout that respects maxSurge and maxUnavailable. There is a tradeoff: you get more rollouts than before, one per config change, and if your ConfigMap has values that churn (feature flags, dynamic settings), you should move those out into a separate ConfigMap that pods read at runtime rather than at start.

We have written the broader pattern for boot-time versus runtime configuration in our Kubernetes stabilization notes, and if this is landing during or right after a migration, the same race appears with any credential source that gets templated as a hook, including some migration recovery playbooks we have run this year.

FAQ

Questions on-call keeps asking after this fires

Can I use kubectl rollout restart safely if the ConfigMap is now correct?

Yes, but only if you have the capacity to churn every pod and your database can absorb the reconnect storm. On a service with 12 replicas and a Postgres max_connections of 200 with pgbouncer in transaction mode, we have done full rollout restarts without incident. On a service that runs hot on connections, do the surgical delete instead.

Why does helm upgrade --atomic not prevent this?

--atomic rolls the release back when the upgrade fails, but the window has already opened by then. The pre-upgrade hook deletes and recreates the ConfigMap before any workload change is applied, so a pod can boot blank while the upgrade is still in flight. The rollback --atomic triggers then runs pre-rollback hooks only, which this ConfigMap is not annotated for, so it leaves the object untouched. --atomic is not a fix for this; removing the hook annotation is.

Does this affect Secret-sourced env the same way?

Yes, identically. envFrom on a Secret has the same start-time-only resolution. If you template Secrets as Helm hooks (some charts do, to inject generated passwords) you get the same race. Same fix: regular resource plus checksum annotation.

How do I know if the race actually fired versus something else broke my pods?

Two-pod comparison. Take one healthy pod and one crashing pod from the same ReplicaSet. If their runtime env for the same key differs (one has DATABASE_URL populated, the other has it empty), it is the ConfigMap race. If both have the same env and one still crashes, look elsewhere: image drift, node-local state, or a downstream dependency.

Can I use a mutating webhook or Reloader to auto-restart on ConfigMap changes instead of the checksum annotation?

Stakater Reloader works and we run it in several client clusters. The checksum annotation is simpler because it is part of the chart and needs no extra controller. If you already run Reloader for other reasons, use it. If you do not, do not install a controller to solve a one-line templating fix.

Getting the surgical recovery right the first time

If you are staring at CrashLoopBackOff right now

The hard part of this recovery is not the kubectl commands. It is deciding which pods to touch when your dashboards are red and the pressure is to do the biggest hammer available. A rollout restart in the middle of a partial outage on a service under connection pressure is how the two-hour incident becomes the six-hour incident. The judgment call is: verify config, inspect env pod-by-pod, delete only what is actually broken, and only then fix the chart so this cannot recur.

At InfraForge we do this recovery with the on-call team on the bridge, in one working session, and we leave the chart patch and the checksum annotation merged before we sign off. If you would rather not run the surgical restart on production by yourself, book a 60-minute infrastructure review and we will walk it with your team today or tomorrow, chart fix included.

Originally published at https://infraforge.agency/insights/helm-rollback-configmap-race-safe-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• Some runs crash at boot with redis.exceptions.ConnectionError: Error 111 connecting to redis:6379. Connection refused. and others start cleanly
• result.json is written on some runs, missing on others, sometimes present but with 0 processed jobs and no error in the log
• Restarting the worker alone (without the backend) makes the problem go away for a while, then it comes back after the next deploy
• A grep for KeyError shows sporadic KeyError: 'job_id' traces that are being caught by a broad except block
• The worker container's uid is 1000, /app/output/ is owned by root:root with mode 0755, and nobody remembers why

The symptom, and what it usually is not

Three failure modes wearing one costume

The reported symptom is almost always the same sentence: 'the worker is flaky, sometimes it processes jobs and sometimes it doesn't, and the logs don't really say why.' That sentence hides three separate bugs that compound. We rank them by the order we actually find them, not by how loud they are in the log:

• 1. Startup race. Worker container comes up before Redis (or the backend that populates Redis) is accepting connections. Retries are missing or set to a value that gives up in under 2 seconds. Roughly 60% of the incidents we see.
• 1. Environment variable drift. The backend sets REDIS_URL, the worker reads CACHE_URL, and the worker's client library silently defaults to redis://localhost:6379. It 'connects' to nothing and hangs or reads empty queues. Roughly 25%.
• 1. Schema skew after a rename. Someone renamed job_id to id on the producer side and missed one .get('job_id') on the consumer. .get returns None, the worker's outer except Exception swallows the downstream KeyError, the job is skipped, no error is logged. Roughly 15%, and the hardest to see.

The cause everyone blames first, and which is almost never it: 'Redis is slow' or 'the worker needs more memory'. We have not once found this to be the actual cause in this failure shape. If you are already sizing up the Redis instance, stop and read the next section first.

There is also a fourth cause that shows up as a hard PermissionError: [Errno 13] Permission denied: '/app/output/result.json' when the worker finally does try to write. It is real, but it is loud. This guide is about the quiet failures.

The discriminating check: read the logs before you touch anything

One grep that ranks the three

Before you restart, redeploy, or scale anything, capture the current state. Restarting the worker throws away the evidence that tells you which of the three you are dealing with. On docker compose, snapshot both services' logs to disk. On Kubernetes, capture kubectl logs --previous for the last crashed worker AND kubectl describe pod for its events. Do this first.

# capture before you touch anything
docker compose logs --no-color --timestamps worker  > /tmp/worker.log
docker compose logs --no-color --timestamps backend > /tmp/backend.log

# k8s equivalent
kubectl logs -n jobs worker-7c9d8f5b6-x2k4m --previous > /tmp/worker.log
kubectl describe pod -n jobs worker-7c9d8f5b6-x2k4m       > /tmp/worker-describe.txt

# then run the discriminating grep
grep -E 'Connection refused|CACHE_URL|localhost:6379|KeyError' /tmp/worker.log | head -40

Snapshot first, grep second. Anything that recreates the container (docker compose up --force-recreate worker, or a down followed by an up) discards the previous container's stdout under the default json-file driver. A plain restart keeps the log file, but it costs you the live process state that made the failure reproducible.

The grep output tells you which cause you have, in this order:

• If you see Error 111 connecting to redis:6379. Connection refused in the first 5 seconds of the worker's log and never again after that, it is the startup race. The worker gave up before Redis was ready.
• If you see Connecting to redis://localhost:6379 (note: localhost, not the service name), or you see NO connection log at all and the queue depth reads always return 0, it is env drift. The worker never got the right URL and its client defaulted.
• If the connection logs are clean, the worker is clearly consuming messages, but the count of 'processed' log lines is lower than the count of 'received' lines, and you can find any KeyError (even one, even caught) in the traces, it is schema skew. The worker is silently dropping jobs whose payload does not match its expected shape.

Order matters. The race hides the drift, and the drift hides the schema skew. Fix them in the order they surface.

The safe fix sequence for each cause

Fix least-destructive first

Fix each cause with the smallest change that discriminates against the others. Do not batch the three fixes into one PR; you will not know which one worked, and the next incident will look identical. Ship them separately, verify each one, then move on.

One thing to name explicitly: the tempting single-line fix of adding sleep 15 to the worker's entrypoint 'solves' the startup race in staging and hides all three bugs in production. We have watched this exact patch get merged, celebrated, and then paged the same team six weeks later when Redis restarted during a maintenance window and the sleep was not long enough. The cost of the right fix (a healthcheck plus retry) is roughly 40 lines of yaml and one afternoon. Pay it.

# docker-compose.yml, the readiness gate that actually works
services:
  redis:
    image: redis:7.2-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 2s
      timeout: 1s
      retries: 15

  backend:
    depends_on:
      redis:
        condition: service_healthy

  worker:
    depends_on:
      backend:
        condition: service_started
      redis:
        condition: service_healthy
    environment:
      REDIS_URL: redis://redis:6379/0    # one name, both sides
    # no sleep, no CACHE_URL fallback

The gate is condition: service_healthy plus a real healthcheck on the dependency. condition: service_started alone only waits for the container to exist, not to be ready.

Confirm the fix by running the stack from cold at least ten times in a row and checking that the processed count equals the received count on every run. If nine out of ten pass, you have not fixed it; you have improved it. Race conditions do not get 90% fixed. For deeper patterns on this kind of intermittent K8s failure, we have written up Kubernetes release failure recovery separately.

FAQ: variants readers usually ask right after

The questions that come next

Is it safe to just add restart: always and let the worker crash-loop until Redis is up? It works, but it makes your logs noisier, it costs you real seconds on every boot, and it hides the underlying dependency graph from anyone reading the compose file later. Use a healthcheck. Save restart: always for actual transient failures in steady state.

Does this apply to RabbitMQ, NATS, or Kafka workers too? Yes, the shape is identical. The verbatim error string changes (Connection refused on RabbitMQ, dial tcp: connection refused on NATS, NoBrokersAvailable on Kafka), but the three causes and the fix order are the same. The env var drift bug is especially common on Kafka clients because KAFKA_BROKERS vs BOOTSTRAP_SERVERS is a coin flip in the ecosystem.

Can I skip the healthcheck if I use Kubernetes with readiness probes? No, readiness probes gate traffic to a pod, not startup order between pods. You still need an initContainer or an application-level retry loop for the worker to wait on its dependency. Readiness alone will not save you.

Why not just fix the broad except Exception: pass and call it a day? Because removing it in isolation will surface the schema skew as a hard crash on production traffic, and if you have not fixed the startup race first, you will not be able to tell which crash is which. Fix in the order the causes appear at boot: connectivity, config, contract.

How do I stop this from recurring? Three things, in order of ROI: (1) a boot-time assertion in every service that fails fast if a required env var is missing or resolves to localhost, (2) a schema check at the consumer boundary that rejects malformed messages loudly instead of silently, (3) a pre-merge integration test that starts the stack from cold and asserts processed == received on 100 test jobs. That third one is the single highest-value test we recommend for job-processing systems.

If you are staring at an empty result.json and the clock is running

When the worker is dropping jobs right now

What makes this class of failure genuinely hard is not any single one of the three bugs. It is that they compound: the race gives you enough noise in the log that the drift looks like a symptom of the race, and by the time you fix the race, the drift has been silently corrupting queue state for hours, and the schema skew is dropping the recovery jobs you are firing to catch up. Untangling that under time pressure is where teams get stuck at 2 in the morning.

We have spent a lot of engineering hours in exactly this shape of incident, on Redis, RabbitMQ, and Kafka, across docker compose and Kubernetes. The pattern above is what we run on the call: capture logs first, grep to rank the causes, fix in dependency order, verify with cold-start runs. If you want a second set of eyes on it while it is happening, book a same-day infrastructure review and we will get on a bridge with your on-call engineer inside a few hours and work the sequence together. If the fires are already out and you want the pre-merge integration test and boot-time assertions in place before the next one, our platform reliability engagements cover exactly that.

Originally published at https://infraforge.agency/insights/worker-drops-jobs-intermittently-startup-race-env-drift/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• NAT gateway data-processing charges (NatGateway-Bytes) climb week over week with no code or workload change

• VPC Flow Logs show heavy traffic to S3 IP ranges (52.216.0.0/15, 3.5.0.0/16) hitting the NAT's ENI instead of the gateway endpoint

• aws ec2 describe-route-tables on a private subnet returns no route for the S3 managed prefix list (pl-xxx)

• NAT gateway BytesOutToDestination stays high while VPC Flow Logs show traffic to the S3 prefix list still routing through NAT (the S3 gateway endpoint itself emits no CloudWatch metrics to check)

• The S3 gateway endpoint exists in the VPC, but the bill still shows $0.045/GB on service-to-service S3 traffic

$412 to $2,103 on a flat workload, all of it NatGateway-Bytes

The bill line item that should not have existed

The NatGateway-Bytes cost for us-east-1 climbed from $412 in March to $2,103 in April. Nothing else on the bill moved. Egress was flat. EC2 was flat. RDS was flat. The whole delta was NAT data processing at $0.045 per GB.

Cost Explorer's usage-type breakdown confirmed the shape. The entire spike was NAT-Bytes on one VPC, in one region, over a six-week ramp. No cliff, no sudden jump, just a slow gradient upward that nobody watched because 'NAT costs a few hundred bucks' was our mental default.

So we mapped the destinations. AWS publishes the IP ranges for S3, DynamoDB, and every other service in ip-ranges.json. We pulled an hour of VPC Flow Logs, joined destination IPs against those ranges, and found the answer. S3 was 71% of the NAT-processed bytes. That should have been impossible. We had a gateway endpoint for S3, and gateway endpoints are free. Their entire point is to keep S3 traffic off NAT.

Cross-region buckets, then a hardcoded public endpoint, then the truth

What we thought first, and why the first two theories died fast

The first theory was that a new batch job was writing to a bucket in a different region. Cross-region S3 traffic does not hit the same-region gateway endpoint; it goes out through NAT. Reasonable theory, wrong theory. We grepped Terraform for cross-region bucket references and found nothing. We checked CloudTrail for recent bucket creations. Nothing new. All our S3 traffic was to buckets in the same region as the workload.

The second theory was that an application was talking to S3 via the public global endpoint URL instead of the regional one. That is a real failure mode: SDKs pointed at a hardcoded https://s3.amazonaws.com sometimes route to the wrong region and skip the gateway endpoint. Also wrong. Every service in the VPC used the SDK default resolver, which picks the regional endpoint.

The third theory turned out to be right. The route table for one of our private subnets was missing the S3 gateway endpoint's prefix-list association. Any pod scheduled onto a node in that subnet was reaching S3 through the NAT, at $0.045 per GB, for six weeks. Same workload the whole time. Different route table.

The route that has to exist, and the day the migration deleted it

How a gateway endpoint quietly stops covering a subnet

A gateway endpoint is not attached to your subnets the way an interface endpoint is. There is no ENI. There is no private DNS record. The gateway endpoint lives in the VPC, and to make it work for a subnet, you have to add its managed prefix list (pl-63a5400a for S3 in us-east-1) as a route in that subnet's route table, with the endpoint as the target.

The whole 'your S3 traffic bypasses NAT' behavior depends entirely on that route existing. If the route is not there, S3 traffic falls through to the default route (0.0.0.0/0), which points at the NAT gateway. There is no error, no warning, no CloudWatch alarm. The traffic just goes through NAT and gets billed per GB.

We checked all four of our private subnets' route tables for the S3 prefix-list route:

aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=subnet-0abc123" \
  --query 'RouteTables[].Routes[?DestinationPrefixListId==`pl-63a5400a`]'

Repeat for each private subnet. Empty result means the S3 gateway endpoint is not covering that subnet.

Three subnets returned the S3 prefix-list route. One returned an empty array. That subnet's route table had been rewritten six weeks earlier during a network migration for another team's project. The migration rebuilt the route table from Terraform, but that Terraform module did not know about the gateway endpoint. The endpoint's association was managed by a separate stack. The rewrite silently dropped the S3 route entry.

Nothing failed. Nothing paged. The workload kept running. It just started paying NAT rates for every S3 GET and PUT the pods on those nodes made.

modify-vpc-endpoint, not create-route

The fix, and the command people reach for that does not apply here

The instinct might be to reach for aws ec2 create-route --route-table-id --vpc-endpoint-id to add the endpoint to the table by hand. That does not fit here. Interface endpoints do not use routes at all (they are ENIs with private DNS), and a gateway endpoint's prefix-list route is not added with create-route either. Gateway endpoints get added to a route table by modifying the endpoint itself and telling it which route tables to associate with.

The correct fix is one call:

aws ec2 modify-vpc-endpoint \
  --vpc-endpoint-id vpce-0abc12345 \
  --add-route-table-ids rtb-0def67890

AWS writes the prefix-list route into the route table atomically. No second command.

Verification took thirty seconds:

aws ec2 describe-route-tables --route-table-ids rtb-0def67890 \
  --query 'RouteTables[].Routes[?GatewayId==`vpce-0abc12345`]'

Should now return the pl-xxx prefix-list route with the endpoint as GatewayId.

The route was there. We then pulled a fresh five-minute slice of VPC Flow Logs and joined against S3's IP ranges again. S3 destinations were dropping out of the NAT sample. The subnet was routing them to the gateway endpoint instead. Over the next hour we watched the NAT gateway's BytesOutToDestination CloudWatch metric drop about 40% and level off.

That was the whole fix. Six weeks of overspend, roughly $1,700 in avoidable NAT charges, closed in about four minutes of actual work.

Endpoint-to-route-table coupling in Terraform, plus a rate-of-change NAT alarm

The two things we changed so this stops happening

We changed two things. First, every gateway endpoint in our Terraform is now paired with an explicit list of route-table IDs it associates with, and that list is generated from the same module that generates the private subnets. When someone adds a subnet, the endpoint's association is derived from the same variable. There is no second stack to remember.

Second, we set up a CloudWatch alarm on the NAT gateway's BytesOutToDestination metric with a per-VPC baseline. Not a fixed dollar threshold, a rate-of-change alarm: if NAT bytes for a VPC exceed three times the trailing seven-day median for two consecutive hours, we get paged. We would have caught this ramp on day four instead of week six.

We considered AWS Cost Anomaly Detection. It does catch this shape of spike, and in our archived bills it did fire, eleven days into the ramp. Our own CloudWatch alarm fires in hours because NAT bytes are a real-time metric and the cost bill is not.

For teams reading this with a similar VPC layout, the audit is roughly a ten-minute job. Enumerate every gateway endpoint, enumerate every route table that should be covered, and check for the prefix-list route in each. If you want the deeper cleanup pattern we use for accumulated cloud-cost drift, we have written more of that up in the InfraForge services overview.

This kind of drift does not fail loudly. It just bills.

If your NAT bill just went sideways and nobody deployed anything

The specific shape of this problem is one people miss because it does not fail loudly. Gateway endpoints have no failure mode that produces an error. They just silently stop covering a subnet, and the bill grows. If you have not audited your route tables against your gateway endpoints in the last six months, there is a decent chance one of your subnets is quietly paying NAT rates for S3 or DynamoDB traffic right now.

We have seen this pattern three times this quarter. Two were the same shape as ours: a route-table rewrite that dropped the prefix-list route. One was a subnet added later that never got the association at all. Each was a five-figure annual overrun that took an afternoon to identify and minutes to fix.

If your NAT gateway bill jumped and nobody deployed anything, book an infrastructure review with our team and we will start with a 30-minute diagnostic call this week. We will pull an hour of your Flow Logs, join destination IPs against AWS service ranges, and tell you which subnet is bleeding before the call ends.

Originally published at https://infraforge.agency/insights/nat-gateway-cost-spike-missing-vpc-endpoint-route/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• Your AWS bill for EKS Compute jumps without a matching traffic or deploy event

• kubectl get nodes returns dozens more nodes than the working set, most at low CPU utilization

• Cluster autoscaler logs repeat 'cannot delete: pdb blocking' and keep naming the same PDB

• A single PodDisruptionBudget in the cluster shows ALLOWED DISRUPTIONS = 0 while others are 1 or higher

• A workload somewhere in the cluster has a pod stuck in ImagePullBackOff that nobody is watching

The $9,800 bill and 60 nodes at 8% CPU

The Friday morning spike

The bill spike was six days old by the time the AWS Budgets alert fired. It tripped at $9,000 for the billing period; the same window the previous month was $4,200. Nothing had shipped that week, CI had been quiet since Wednesday afternoon, and no product team was asking for extra capacity.

Compute Optimizer had already flagged the cluster's managed node group as significantly overprovisioned. kubectl get nodes returned 60 m5.4xlarge nodes; the normal working set for this cluster is 20. Forty of the extras were sitting under 8% CPU, hosting kube-system DaemonSet pods and a single Deployment nobody had touched in months.

The math is straightforward. m5.4xlarge on-demand in us-east-1 is $0.768 an hour. Forty extra nodes for six days is about $4,400 of surprise burn, most of the jump from the $4,200 baseline. The overnight autoscale had happened on the previous Saturday to run a scheduled batch job; the batch finished by 6 am, and then the cluster autoscaler just stopped scaling down. Six days of that pattern is how a bill doubles.

Why HPA runaway and traffic burst were both wrong

Two theories that died fast

Two theories came up on the bridge in the first five minutes. The obvious one was HPA runaway: a Horizontal Pod Autoscaler misreading its metric and scaling a Deployment to hundreds of replicas, forcing the cluster autoscaler to hold capacity to place them. The second was a traffic burst: some overnight event pushing real request volume up, driving replicas up, then subsiding but leaving the nodes.

Both died fast. kubectl get hpa --all-namespaces showed every HPA sitting comfortably below its ceiling, none within striking distance of a scale trigger. Prometheus request-rate graphs for every ingress-fronted service were flat across the whole week. When we summed pod counts across the cluster, we got 342, roughly what a normal Friday looks like, and nowhere near the 900+ pods it would take to justify 60 m5.4xlarge nodes at typical density.

The demand side was fine. Something on the supply side was refusing to shrink.

The cluster-autoscaler log that pointed at exactly one PDB

One PDB, named in every log line

We went to the cluster autoscaler's own logs, which is where CA tells you plainly why it will not do the thing you want. kubectl -n kube-system logs deploy/cluster-autoscaler --tail=200 returned the same shape of line, forty times per scale-down cycle, each naming a specific node, all of them naming the same PDB.

I0313 09:32:04.881 static_autoscaler.go:428] Scale down status: lastScaleDownFailTime=2026-03-13 09:22:04
I0313 09:32:04.892 cluster.go:151] Fast evaluation: node ip-10-42-14-88.ec2.internal, cannot delete: pdb data-platform/nightly-rollup-pdb blocking eviction of pod data-platform/nightly-rollup-7f8b9c5d4-4kx2m (currentHealthy=39, desiredHealthy=39)
I0313 09:32:04.895 cluster.go:151] Fast evaluation: node ip-10-42-15-19.ec2.internal, cannot delete: pdb data-platform/nightly-rollup-pdb blocking eviction of pod data-platform/nightly-rollup-7f8b9c5d4-9h4tn (currentHealthy=39, desiredHealthy=39)
I0313 09:32:04.897 cluster.go:151] Fast evaluation: node ip-10-42-15-33.ec2.internal, cannot delete: pdb data-platform/nightly-rollup-pdb blocking eviction of pod data-platform/nightly-rollup-7f8b9c5d4-b7q9r (currentHealthy=39, desiredHealthy=39)
... (37 more lines, all identical shape, all naming data-platform/nightly-rollup-pdb) ...
I0313 09:32:05.114 scale_down.go:1052] Scale down considered 40 nodes, 0 removable

Every blocked node in every iteration named the same PDB. That was the whole answer in the first log grab.

We cross-checked against the cluster's full PDB inventory to confirm no other PDB was involved.

$ kubectl get poddisruptionbudgets --all-namespaces
NAMESPACE       NAME                     MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
api-gateway     api-gateway-pdb          2               N/A               3                     47d
auth            auth-service-pdb         1               N/A               2                     47d
data-platform   nightly-rollup-pdb       39              N/A               0                     213d
data-platform   warehouse-writer-pdb     1               N/A               2                     47d
payments        payments-api-pdb         2               N/A               1                     47d
search          search-indexer-pdb       1               N/A               2                     47d

Six PodDisruptionBudgets. Exactly one at zero allowed disruptions. Every other PDB had two or three to spare, which is the healthy state.

The outlier was data-platform/nightly-rollup-pdb, and it was 213 days old, while the other five had all been re-created seven weeks earlier as part of a cluster upgrade. Something about this specific PDB had been quietly wrong for a long time.

We looked at the underlying workload. The 'nightly-rollup' service started life as a CronJob and had been reimplemented at some point as a 40-replica Deployment with soft pod anti-affinity, running continuously to feed a downstream aggregation service. Its PDB carried minAvailable: 39, which is the standard 'allow one disruption at a time' setting for rolling updates. Fine when the workload is healthy.

kubectl get pods -n data-platform -l app=nightly-rollup told the rest of the story. Thirty-nine pods Running, one pod stuck in ImagePullBackOff. Six days earlier, someone had pushed a rolling update with a container image reference that pointed at an ECR registry we had migrated away from during a project the previous quarter. Every other workload had been re-pointed at the new registry during that migration; this one Deployment had been missed, and nobody had noticed because it 'just worked' for months.

The instant one pod dropped out of Ready, healthy went from 40 to 39 against a minAvailable floor of 39, and allowedDisruptions clamped to zero. From that moment on, no node hosting a nightly-rollup pod could be drained: evicting one would take healthy below the floor, and cluster autoscaler is careful about that. Forty pods, forty nodes, one stuck PDB. Each nightly-rollup pod requested about 400m CPU and 1.5 GiB of memory, which is a rounding error on an m5.4xlarge, but the pod's presence plus the PDB at zero meant the node was unremovable regardless of how much headroom the rest of the box had. Forty nodes at 5-8% CPU, all of them technically pinned by one broken image reference.

Why we deleted the PDB instead of pushing the fixed image

Delete the PDB, or fix the image

Two paths out. Fix the underlying pod: patch the Deployment's image reference, let the rolling update roll forward, healthy count returns to 40, allowedDisruptions goes positive, CA is free. Or delete the PDB directly: no PDB, no eviction block, CA can drain nodes.

We almost went with the image fix, because that is the 'correct' answer in an abstract sense; the PDB is doing what it was written to do. The problem was who owned the Deployment. The team that had originally built the nightly-rollup service had been reorganized eighteen months earlier, and the current owning team had inherited it in a spreadsheet handoff and had never shipped a change to it. To fix the image correctly we needed either to push the correct image to the old ECR registry (which required someone with write access on an account we were sunsetting) or to patch the Deployment to point at the new registry (which required knowing whether the image tag existed there and had feature-parity, which we did not, at 9:30 am).

The PDB, on the other hand, was clearly orphaned. Its author was gone. Its minAvailable: 39 was arithmetically fine for a 40-replica Deployment during healthy operation but it meant one broken pod would freeze the entire fleet, which is exactly what had happened. Deleting the PDB did not remove any replicas or affect the currently-running pods; it just removed the eviction guardrail.

kubectl delete poddisruptionbudget -n data-platform nightly-rollup-pdb
# poddisruptionbudget.policy "nightly-rollup-pdb" deleted

One command. The gate is the PDB; remove the gate.

Twelve minutes later, cluster autoscaler started removing nodes. It took about ninety minutes to work through the fleet. The nightly-rollup pods that got evicted rescheduled onto the remaining nodes without issue because the anti-affinity was preferredDuringSchedulingIgnoredDuringExecution (soft), which meant CA could consolidate the fleet onto fewer nodes when memory allowed. The pending pod stayed pending, because the underlying image reference was still wrong, but a single pending pod is a monitoring problem, not a bill problem.

By 11:30 am the cluster was back at 24 nodes. We paged the data team's on-call to fix the image reference at their leisure the following Monday. The bill for that Friday's Compute came in at $760, back inside the normal envelope.

The instinct in the first thirty minutes had been to kubectl delete node on the idle boxes. That would not have worked, and not for the reason you might expect. kubectl delete node only drops the Node object from the API server; the kubelet on that instance re-registers within seconds and the node comes back with the same name. The EC2 instance is never terminated and the orphaned PDB is untouched, so the spend does not move. Kicking nodes was never going to touch this. The PDB is the gate. Remove the gate or fix the gated pod; kicking the nodes directly is a way to spend money without changing anything.

The Kyverno policy and PDB alert that closed the gap

The rules we shipped the next week

The postmortem produced three changes we shipped inside the week.

The first was a cluster-level admission policy that rejects new PDBs unless they carry an owner annotation and an expires-at annotation. If a PDB is going to have the power to freeze half a cluster, someone needs to own it and someone needs to argue for renewing it. Kyverno was already installed for other reasons, so the rule was about thirty lines.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: pdb-lifecycle-required
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-owner-and-expiry
      match:
        any:
          - resources:
              kinds:
                - PodDisruptionBudget
      validate:
        message: "PDBs must carry infraforge.io/owner and infraforge.io/expires-at annotations. See runbook: https://internal.infraforge.io/rb/pdb-lifecycle"
        pattern:
          metadata:
            annotations:
              infraforge.io/owner: "?*"
              infraforge.io/expires-at: "?*"

Kyverno ClusterPolicy blocking any PDB that arrives without an owner or an expiry.

The exemption path for genuine long-lived PDBs (data-plane primary databases, for example) is to set infraforge.io/expires-at to a far-future date and put the owning team alias in infraforge.io/owner. That does not prevent the specific failure we hit, but it does mean that in eighteen months when the next team reorganization happens, we can grep for the PDBs whose owners no longer exist before they become orphaned.

The second change was a Prometheus alert that fires when any PDB has allowedDisruptions == 0 for more than thirty minutes. Not five, not sixty. Thirty is long enough that a healthy rolling update has finished but short enough that a stuck one still gets caught inside the same business day.

The third was a workload audit that the source scenario also recommends. Over the following week we ran kubectl top pods on a rolling seven-day window against every Deployment with minReplicas > 1, and dropped the floor on five workloads whose real utilization was under 30%. Two of them we took from minReplicas: 3 to minReplicas: 1. That work did not fix the incident (the PDB was the actual bug), but it lowered the cluster's headroom baseline enough to knock another two nodes off the normal working set.

On the observability side we replaced the flat AWS Budgets threshold with anomaly detection at the EKS Compute line item, tuned to fire on a 30% week-over-week jump. The flat $9,000 threshold caught this incident six days late. The anomaly rule would have caught it inside twenty-four hours.

When your EKS bill doubles without a traffic event

When this is happening to you

Bill spikes without a matching traffic event almost always trace to a controller doing exactly what it was configured to do, in a way the config never anticipated. Cluster autoscaler is the most common source of this specific flavor, because its downscale logic is deliberately conservative around PodDisruptionBudgets; a single PDB with zero allowed disruptions can pin nodes indefinitely, and if the PDB is stuck on a pod nobody currently owns, nobody notices until the bill does. HPA misconfiguration, Karpenter provisioner drift, and stuck node-termination lifecycle hooks are the other three variants we see most often in the same shape.

We run recovery engagements with this exact shape most quarters. The fastest path to the answer is usually the cluster autoscaler logs, not a Prometheus dashboard, because CA will just tell you what it is refusing to do. If your EKS bill doubled inside a week and Compute Optimizer is flagging the node group as overprovisioned, we can be on a bridge with your platform team the same day. Book an infrastructure review and we will start with a 30-minute diagnostic call this week; the more artifact you can share ahead of the call (bill line item, kubectl get nodes output, and a kubectl -n kube-system logs deploy/cluster-autoscaler --tail=500 grab), the faster we can name the specific blocker.

If you are earlier in the shape (rising EKS bill, no obvious cause yet, no acute page), the Kubernetes and CI/CD stabilization work we do covers the workload-audit and admission-policy side of what this article described. We have written more on the general pattern of cost spikes driven by controller behavior in the cloud cost spikes problem write-up.

Originally published at https://infraforge.agency/insights/eks-autoscaler-stuck-pdb-cost-spike/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• Application logs 'schema version N found, expected M' and exits before serving traffic
• Helm rollback to the previous chart version fails with the same or inverse schema error
• A migration job pod shows exit code 137 or OOMKilled in kubectl describe
• The schema_migrations (or equivalent) table reports a version the table DDL does not actually match
• Restoring from the most recent Postgres backup would lose hours of production data the team needs to keep

Both the old and new binary refused to start against the same database

The log line that ruled out a rollback

The on-call had done the obvious thing first. The new chart was failing, so they ran helm rollback to the previous revision. The previous revision's pod came up, hit the database, and crashed too, for a different reason. The new binary expected schema 23 and found the version row marked dirty mid-migration. The old binary expected 21 and found the row already advanced to 23. One saw a migration that never finished, the other saw a database from the future. Both were sort of right.

That pattern is what told us the database was the problem, not the chart. A clean rollback should have produced a running pod. If both binaries reject the same database, the database is not in either of the states they expect. It is in a third state nobody coded for.

$ kubectl logs -n statuspage statuspage-app-7b9f-xq2vk
INFO  starting statuspage v0.90.78
INFO  connecting to postgres at postgres.statuspage.svc:5432
ERROR database schema version 23 found, expected 21
FATAL refusing to start with schema version mismatch

$ kubectl rollout history deployment/statuspage-app -n statuspage
REVISION  CHANGE-CAUSE
14        helm upgrade statuspage statuspage/statuspage --version 0.91.2
15        helm rollback statuspage 14

$ kubectl logs -n statuspage statuspage-app-6c4d-7m9pz   # rolled-back pod
ERROR database schema version 23 found, expected 21
FATAL refusing to start with schema version mismatch

Different errors from the two chart revisions, same root cause: the database, not the chart, was the problem.

The version row claimed 23. The table DDL was somewhere between 22 and 23.

What the schema_migrations table actually said

We dropped into psql against the application database and pulled the migration tracking table. The row said version 23, applied. The dirty flag was true, which on most migration libraries means 'a migration started running and never reported success'. That single boolean was the thread we pulled on for the next hour.

statuspage=> select * from schema_migrations;
 version | dirty
---------+-------
      23 | t
(1 row)

statuspage=> \d incidents
          Table "public.incidents"
   Column    |  Type   | Nullable | Default
-------------+---------+----------+---------
 id          | bigint  | not null |
 service_id  | bigint  | not null |
 started_at  | timestamp without time zone |          |
 resolved_at | timestamp without time zone |          |
 title       | text    |          |
-- expected per migration 0023: severity column, incident_updates FK, partial index on resolved_at IS NULL
-- present: none of the above

The version row was lying. The table structure was a partial 23.

We pulled the migration files out of the chart's image and read them. Migration 0023 was three statements: add a severity column, create an incident_updates table with a foreign key back, create a partial index on unresolved incidents. None of the three were present in the live schema. The OOM had hit after the migration library wrote the version row and before any DDL statement actually committed. Or possibly between statements. The order was implementation-specific and we did not care which exact moment because the answer was the same: none of 0023's DDL had landed, but the bookkeeping said it had.

This is the specific failure mode that makes partial migrations dangerous. The migration library and the actual schema disagree, and the application trusts the library. The library trusts a row it wrote in a different transaction than the DDL it was supposed to be tracking. Most migration tools have fixed this since 2020 by wrapping version-row-and-DDL in a single transaction, but a long tail of applications still ship with the older split-transaction behaviour, and you only find out which one you have when something interrupts a migration.

The base backup was 6 hours stale and uptime data is the product

Why we did not restore from backup

The instinct, and the safe move on most days, is to restore Postgres from the last known-good base backup and replay WAL up to a point just before the migration started. We checked the backup. It was a nightly pg_basebackup, 6 hours old, and WAL archiving had been configured but never tested for PITR. We could probably have done it. We were not willing to bet the status page on 'probably' while the status page was already down.

More importantly, the uptime check history is the product. A status page that loses 6 hours of check data after an outage is worse than a status page that takes another hour to come back. We talked it through with the team lead and decided the database in front of us was recoverable, and recovering it was lower risk than the restore path. That decision is worth naming because it goes against the usual 'just restore from backup' instinct. When the data itself is the value, finishing a half-migration by hand is often the right call.

Three DDL statements, a version-row correction, and a careful restart

Finishing the migration by hand

We pulled migration 0023 verbatim from the chart image, rewrote each statement with IF NOT EXISTS guards so a re-run could not double-apply, and ran them inside a single transaction so any failure left the database where we found it. Before touching anything we took a pg_dump of the application schema and data to a local file. That dump was our 'we can always undo this' insurance, separate from the production backup system.

-- 1. snapshot first, outside any transaction
$ kubectl exec -n statuspage postgres-0 -- \
    pg_dump -U statuspage -Fc statuspage > /tmp/statuspage-pre-repair.dump

-- 2. complete migration 0023 inside one transaction
BEGIN;

ALTER TABLE incidents
  ADD COLUMN IF NOT EXISTS severity smallint NOT NULL DEFAULT 3;

CREATE TABLE IF NOT EXISTS incident_updates (
  id           bigserial PRIMARY KEY,
  incident_id  bigint NOT NULL REFERENCES incidents(id) ON DELETE CASCADE,
  body         text NOT NULL,
  created_at   timestamp without time zone NOT NULL DEFAULT now()
);

CREATE INDEX IF NOT EXISTS incidents_unresolved_idx
  ON incidents (started_at)
  WHERE resolved_at IS NULL;

-- 3. clear the dirty flag, version row already says 23
UPDATE schema_migrations SET dirty = false WHERE version = 23;

-- 4. sanity-check before commit
SELECT version, dirty FROM schema_migrations;
\d incidents
\d incident_updates

COMMIT;

All four statements committed in 180ms: the three DDL statements plus the version-row correction. The read-only checks below them commit nothing. The dirty flag was the last thing to flip.

After the commit we scaled the application deployment from 0 to 1 (we had scaled it to 0 earlier to stop the CrashLoopBackOff noise) and watched the pod logs. It connected, read schema_migrations, found 23 clean, started serving. We curled the health endpoint, got a 200, then hit /api/services and confirmed all the configured uptime checks were present with their full history intact. Total time from 'database is the problem' to 'status page is back': 51 minutes.

The thing worth saying out loud about this kind of repair: it works because the migration was small and the failure was clean. If 0023 had been a data migration that rewrote a million rows halfway, the recovery would have looked very different and the backup-restore path would have won. Always read the failed migration before you decide which recovery to attempt. We have written more about that decision in the migration recovery playbook.

A schema snapshot before every migration, and migration jobs that cannot be OOMKilled

The pre-upgrade hook we shipped the next day

Two changes went in within 24 hours. First, the Helm chart now has a pre-upgrade hook that runs a pg_dump of the schema (not the data, just the structure plus schema_migrations row) to an object storage bucket, tagged with the chart version it is about to migrate to. If a future migration goes sideways, the recovery starts from a known structure-level snapshot taken seconds before the migration began, not from the nightly backup. The hook adds about 4 seconds to every upgrade and has paid for itself once already.

Second, the migration job spec got real resource requests and limits, and the limit is now 2x the largest migration we have ever observed in staging plus a 50% buffer. The OOM happened because the chart shipped with a 256Mi limit and the migration that finally hit the production data size needed about 380Mi. We also removed the liveness probe from the migration job entirely. A migration that takes longer than expected should not be killed by Kubernetes; it should be allowed to either finish or fail on its own terms so the migration library can write a clean dirty=true and exit.

The shape of every upgrade now. The branch on the right is the one that saved us.

We have stopped recommending that teams skip the pre-migration schema snapshot just because their database is 'small enough to restore from nightly'. The nightly backup answers a different question than the schema snapshot. The nightly tells you what the data looked like 6 hours ago. The schema snapshot tells you what shape the database was in seconds before this specific migration began, which is the thing you need when a migration is the thing that broke.

Recovering a partial migration without losing the data behind it

When a status page is the thing that is down

The reason this kind of incident is hard is not the SQL. The SQL is usually three or four statements you can read off the migration file. The hard part is deciding whether to finish by hand or restore from backup, and that decision depends on details most teams have not catalogued: how their migration library handles the version row, whether PITR has ever been tested, whether the failed migration is structural or data-rewriting, and whether the data between the last backup and now is recoverable some other way.

We run these recovery engagements regularly. We have seen the OOMKilled-migration-job pattern four times in the last year, two of them on status pages or monitoring tools where the data IS the product, and we have a checklist for the decision now. If you are staring at a CrashLoopBackOff with a schema version mismatch in the logs and a Helm rollback that did not help, book an infrastructure review and we will be on a bridge with you the same day to work through the finish-by-hand versus restore decision before you commit to either.

Originally published at https://infraforge.agency/insights/recovering-status-page-half-finished-schema-migration/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• Apps stuck in Progressing or Degraded for hours after a hardening or security pass, with no single obvious cause in the events stream
• Status ConfigMaps written by an in-cluster reconciler get rewritten within 10 to 15 seconds of any kubectl patch
• kubectl delete job hangs on suspended PreSync Jobs because a hook-cleanup finalizer is still attached
• Migration init containers crash-loop with pg_isready succeeding but a schema verification step failing
• kubectl patch on a RoleBinding fails with 'cannot change roleRef' because roleRef is immutable, so the binding has to be recreated

Why editing the status ConfigMap was the wrong instinct

The patch that survived ten seconds

The team had built a small in-house control plane the year before. A Python reconciler Pod in bleater-system watched the managed workloads, computed health from live cluster signals, and wrote a bleater-status ConfigMap every ten to fifteen seconds. Five apps reported there: an auth service, a profile service, a timeline service, a fanout service, and a primary application that handled the user-facing API. None of them used a full GitOps platform. The reconciler followed GitOps conventions, PreSync hook Jobs, sync windows, hook-cleanup finalizers, but it operated on raw Kubernetes primitives. ConfigMaps, Jobs, Roles. No CRDs.

That detail matters because when the on-call lead patched the status ConfigMap to mark the apps healthy, the reconciler was doing its job. It read the live cluster, saw the upstream signals were still bad, and rewrote the status. The patch was not wrong because patching ConfigMaps is wrong. It was wrong because the bleater-status object was not an input to the system. It was an output. Editing an output to fix a system is the same shape of mistake as editing a Prometheus metric to fix a service.

We have seen this pattern enough times to write it down as a rule. If a controller is rewriting your patches in under a minute, the object you are patching is derived state. Find the inputs. The reconciler source was eighty lines of Python and it took two minutes to read. The health predicate was an AND-chain across eight signals: lock state, orphan hook Job (the finalizer), schema version, RBAC capability, PVC bound, ResourceQuota headroom, NetworkPolicy egress, and init-container health. Any one of the first seven returning bad meant Progressing; a failing init container returned Degraded instead. All eight were bad.

# the AND-chain we found in the reconciler
def app_health(app):
    if lock_status() == 'locked':
        return 'Progressing'
    if orphan_hook_present(app):
        return 'Progressing'
    if schema_declared_version() < required_version():
        return 'Progressing'
    if not migration_rbac_capable():
        return 'Progressing'
    if not pvc_bound(app):
        return 'Progressing'
    if quota_exhausted():
        return 'Progressing'
    if not egress_allows_db(app):
        return 'Progressing'
    if init_container_failing(app):
        return 'Degraded'
    return 'Healthy'

The reconciler's health function. Eight independent signals, all gating. Every patch to the output ConfigMap was wasted work until every signal flipped.

What the inventory pass turned up in the bleater namespace

Eight failures wearing one hat

We started with the inventory, because the ticket told us almost nothing. A real P1 page rarely enumerates faults; it tells you what is on fire and gives you the namespace. We ran the kind of get-everything pass we always run on a strange namespace.

kubectl get pods,configmaps,jobs,deployments,roles,rolebindings,serviceaccounts,pvc,resourcequota,networkpolicy -n bleater
kubectl get events -n bleater --sort-by=.lastTimestamp | tail -40
kubectl describe pod -n bleater | grep -A5 'Init Containers\|Events:'

The first three commands we ran. The namespace had about sixteen pre-existing platform workloads from other teams sharing label values with the five managed apps.

What came back was a layered mess. A suspended PreSync Job named auth-presync-migrate-legacy7r2x with a hook-cleanup finalizer and no hook-delete-policy. A second suspended Job named fanout-presync-validate that looked identical but carried the hook-delete-policy annotation and a bleater.io/owner label pointing at platform-team. A hook-reconciliation-lock ConfigMap with status: locked and a stale lock-reason from the night of the rollout. The primary application's pod in Init:CrashLoopBackOff with kubectl logs --previous showing the init container failing after pg_isready returned ok. A bleat-db-schema ConfigMap declaring version=2 with no tables-v3 key. A migration script that contained psql ... || exit 0 and had no set -e.

And then the governance layer, which is where the rollout had really gotten out of hand. A RoleBinding named migration-runner-binding pointed at migration-runner-role-v1, which had read-only verbs. A migration-runner-role-v2 existed alongside it, unbound, with create:jobs and patch:configmaps. A PersistentVolumeClaim named bleat-migration-pvc was Pending with an event saying storageclass.storage.k8s.io "fast-ssd-tier" not found, on a k3s cluster where the only storage class was local-path. A ResourceQuota set to pods: 1. A NetworkPolicy with egress: [] denying everything outbound including DNS.

Each one of those, taken alone, was a small fix. Taken together, they gated each other. The migration could not run because the RBAC was wrong. The repair Pods could not schedule because the quota was at one. The init container could not reach Postgres because the NetworkPolicy denied egress. The schema could not advance because the script swallowed errors. The reconciler refused to mark anything healthy until all of them resolved. The hardening rollout had tightened every knob at once and the knobs were not independent.

The dependency graph we drew on the bridge call. Cascade order falls out of the arrows.

The order of repair when faults gate each other

Why we raised the quota before anything else

The instinct on a multi-fault incident is to start with the most visible symptom. The CrashLoopBackOff is loud. The lock is loud. The orphan Job is loud. None of those were the right first move. The right first move was the boring one: raise the ResourceQuota, because every other fix needed to schedule a Pod. A ResourceQuota of pods: 1 caps the total number of non-terminal Pods in the namespace, not the number beyond what is already running, and about sixteen were already running. The quota was over-committed the moment the rollout landed it, so the API server admitted nothing new. We raised it to pods: 24, cpu: 8, memory: 16Gi, sized above the existing workloads plus the five managed apps and the repair Pods. Production limits. We did not delete the quota.

Then the RBAC. We described both Roles and confirmed v2 had the verbs the migration Job needed. Patching the existing RoleBinding to swing roleRef to v2 returned the error we expected.

$ kubectl patch rolebinding migration-runner-binding -n bleater \
    --type='json' -p='[{"op":"replace","path":"/roleRef/name","value":"migration-runner-role-v2"}]'
The RoleBinding "migration-runner-binding" is invalid: roleRef: Invalid value: rbac.RoleRef{...}: cannot change roleRef

$ kubectl get rolebinding migration-runner-binding -n bleater -o yaml > /tmp/rb.yaml
# edit /tmp/rb.yaml, set roleRef.name to migration-runner-role-v2
$ kubectl delete rolebinding migration-runner-binding -n bleater
$ kubectl apply -f /tmp/rb.yaml

roleRef is immutable. The only path is delete-and-recreate, with the existing object as a template so you do not lose subjects.

PVC next. We listed storage classes, saw local-path was the only one, exported the existing PVC, changed storageClassName, deleted, reapplied. The PVC sat Pending for a few more seconds until we scheduled a consumer Pod against it, because local-path on k3s binds on first consumer. Then the NetworkPolicy. We did not delete it. The deny-by-default posture was the right posture; the rollout had just forgotten to allow anything. We added three explicit egress rules: same-namespace for the Postgres reach, kube-system on UDP 53 for DNS, and the metrics endpoints the reconciler scraped. The deny-all stayed in place for everything else.

Then the lock and the orphan. The lock was a one-line patch to set status: unlocked and to replace lock-reason with resolved-2024-hardening-rollback. We left an audit value rather than blanking the field. The orphan Job hung on delete because of the finalizer. The strip-then-delete sequence is muscle memory at this point but it is worth showing because plenty of teams reach for --force first, which is the wrong tool.

# strip the finalizer first, then delete cleanly
kubectl patch job auth-presync-migrate-legacy7r2x -n bleater \
  --type=json -p='[{"op":"remove","path":"/metadata/finalizers"}]'
kubectl delete job auth-presync-migrate-legacy7r2x -n bleater

# do NOT touch fanout-presync-validate. it has hook-delete-policy set,
# carries bleater.io/owner=platform-team, and the reconciler manages it.
kubectl get job fanout-presync-validate -n bleater \
  -o jsonpath='{.metadata.annotations.argocd\.argoproj\.io/hook-delete-policy}'
# => HookSucceeded

Strip then delete. The decoy Job looks identical to the orphan from a distance; the discriminator is the hook-delete-policy annotation and the ownership label.

We have written more on cleaning up GitOps-style state safely in our Kubernetes and CI/CD stabilization playbook, including the finalizer-strip pattern and how to tell a managed Job from an orphaned one without guessing.

Don't weaken governance to silence alarms

The fixes that had to be repairs, not deletes

Halfway through the recovery the client's platform lead asked the obvious question. Why not just delete the ResourceQuota and the NetworkPolicy until things stabilize, then put them back? It would have shaved twenty minutes. We said no, and the reason is worth writing down, because it is the part of incident work that teams under pressure get wrong most often.

Governance controls exist for a reason. Someone put pods: 1 on that ResourceQuota originally because something had blown up the namespace before. Someone put the deny-all egress on because the auth service should not be able to call random external endpoints. The rollout had mangled the values, not the intent. Deleting the controls would have restored the workloads and silenced the alarms. It would have also removed two of the few real defenses that namespace had, with no scheduled work item to put them back. We have watched teams do this in March and find the controls still missing in November. The graveyard of post-incident TODOs is full of governance restore tickets that never got worked.

So we repaired. The quota went up to production limits in place. The NetworkPolicy got explicit allow rules added while the default-deny stayed. The PVC got a real storage class while the claim itself stayed at the same name and the same size. The orphan Job got deleted, because a stale suspended PreSync Job genuinely is garbage, but the cascade infrastructure stayed. Same controls, working values.

The migration script was the other repair-not-delete case. The version we found had this pattern:

# what we found
#!/bin/bash
psql -h $DB_HOST -U $DB_USER -d $DB_NAME -f /migrations/v3.sql || exit 0
echo "migration complete"

# what we replaced it with
#!/bin/bash
set -euo pipefail
psql -h $DB_HOST -U $DB_USER -d $DB_NAME \
  -v ON_ERROR_STOP=1 \
  -f /migrations/v3.sql
echo "migration complete"

|| exit 0 is the single worst line in any migration script. set -e and ON_ERROR_STOP=1 together mean a failing SQL statement actually fails the Job, which is what the reconciler was waiting to see.

After the script was patched, the migration Job ran successfully under the new RoleBinding, applied the v3 schema, and we read the tables back out of Postgres directly rather than trusting the script's exit code. The bleat-db-schema ConfigMap got tables-v3 written from observed pg_tables output. Not from the migration's stated intent. From the live database. If you ever find yourself writing schema declarations from anything other than what is actually in the database, you are setting up the next incident.

When in-house reconcilers and hardening rollouts collide

If your control plane is gaslighting your operators

The hard part of this kind of incident is not any single fault. The hard part is that an internal control plane is opinionated about state in ways that are not documented anywhere except in the reconciler's source code. When five apps are red and the dashboard says nothing changed, your team can spend an hour patching outputs that get reverted before they understand the inputs. Hardening rollouts make this worse, because they touch ResourceQuotas and NetworkPolicies and RBAC in the same change window, and the rollback path almost never accounts for the case where the controls themselves were the right idea but the values were wrong.

We run these recovery engagements every week. The in-house reconciler pattern shows up at almost every SaaS company past Series A that decided not to run ArgoCD or Flux directly. The shape of the failure is always the same: a small Python or Go service that watches a namespace and writes a status object, an operations team that does not own the reconciler code, and a control plane that fights every cosmetic fix because that is what it was built to do. We have seen the RoleBinding immutability case four times this quarter alone. The NetworkPolicy egress-without-DNS case shows up after every security audit cycle.

If you are watching a namespace where the status object keeps reverting your changes, or where a hardening pass cascaded across half a dozen layers and your team is debating whether to delete the controls to get back to green, book an infrastructure review with our team and we will be on a bridge call with you the same day. We will read your reconciler, draw the dependency graph for the cascade, and walk the repair order with your on-call. The goal is not to get the dashboard green by morning. The goal is to get it green without leaving a graveyard of governance restore tickets behind it.

Originally published at https://infraforge.agency/insights/internal-control-plane-cascade-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• kubectl patch or apply on a ConfigMap returns a failed calling webhook error (connection refused, or a timeout) instead of a normal validation error
• An admission webhook pod is CrashLoopBackOff while its ValidatingWebhookConfiguration is set to failurePolicy: Fail
• ArgoCD shows sync pending or OutOfSync on resources in the affected namespace and the sync will not progress
• A workload reads stale config from a ConfigMap that was supposedly already updated, and a Deployment-level env var is shadowing the mounted value
• Compliance requires that admission webhooks remain failurePolicy: Fail in production, so flipping to Ignore as a workaround is itself an audit event

We thought it was a credentials incident for the first 20 minutes

The patch that came back as a webhook call failure

The page that started the call was a profile service returning 500s on /health. The cause looked obvious. The data layer team had rotated the MongoDB credentials the day before, and the live ConfigMap in the application namespace still held the old password. There was a backup ConfigMap sitting next to it with the rotated values, labelled exactly the way the runbook described. The fix was supposed to be a thirty second kubectl patch.

It was not. The patch came back with this:

$ kubectl -n app patch configmap profile-mongodb-config \
    --type merge --patch-file rotated.yaml
Error from server (InternalError): Internal error occurred:
failed calling webhook "configmap-validator.app.svc":
Post "https://configmap-validator.app.svc:443/validate?timeout=10s":
dial tcp 10.42.7.83:443: connect: connection refused

The actual error. Not a credentials problem, an admission control problem.

That error string is the whole story. The cluster had a ValidatingWebhookConfiguration named configmap-validator that intercepted every ConfigMap write in the namespace. The webhook pod was supposed to enforce a schema policy that the compliance team owned. Right now the webhook pod was not answering on its service IP, which meant every ConfigMap write was failing closed, which meant our credential fix was failing closed, which meant the profile service stayed down.

We had walked into this kind of shape before, but usually on the cert-manager side. This time the trap was tighter: the webhook was supposed to validate the very ConfigMaps that controlled the workloads in its own namespace, and one of those workloads happened to be down for an unrelated reason. Two independent failures had stacked into a deadlock.

Why the pod was crash-looping and why nothing could fix it in place

The webhook was guarding its own ConfigMap

kubectl describe on the configmap-validator pod told us the liveness probe was failing. The pod was getting killed every 30 seconds, restarting, getting killed again. The probe was hitting /healthz on port 8443. The actual application served its health endpoint on /health, no z. Someone had copy-pasted a probe spec from an older service months ago and nobody had noticed because the webhook had been running fine until a recent image bump shifted the health route.

Fixing a Deployment in Kubernetes is normally a kubectl edit deploy or a kubectl patch on the probe spec and you are done. That was not available to us. The webhook configuration intercepted ConfigMap writes, not Deployment writes, so technically we could have patched the Deployment directly. Except the Deployment mounted a ConfigMap for its own startup arguments, and our platform team had a hard rule against in-cluster edits that drifted from the GitOps source. ArgoCD would self-heal the Deployment back to the broken probe spec inside 90 seconds.

We needed the correct health path, and the compliance team kept the canonical values in a separate namespace. Their ConfigMap held the approved liveness path, the approved annotation policy, and the compliance acknowledgement token that any incident response was required to reference. We pulled it:

$ kubectl -n compliance get configmap webhook-standards -o yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: webhook-standards
  namespace: compliance
data:
  liveness-path: "/health"
  readiness-path: "/ready"
  failure-policy: "Fail"
  ack-token: "COMP-ACK-7c3f9a-2024Q4"
  required-annotations: |
    incident.compliance/id
    incident.compliance/services
    incident.compliance/ack-token

The compliance source of truth. We read these values; we did not retype them.

Flip failurePolicy to Ignore, or delete the ValidatingWebhookConfiguration?

Choosing the smaller blast radius

There were two ways to unblock ConfigMap writes. We could delete the ValidatingWebhookConfiguration entirely, fix everything, and recreate it from the GitOps source. Or we could patch failurePolicy from Fail to Ignore for the length of the recovery and patch it back when we were done.

Option B won because of the audit trail. The compliance team would rather see one field flip and one field flip back, with the same controller object identity across the incident, than see a delete and a recreate with a new resourceVersion lineage. That is the kind of preference you only learn by sitting through an audit. We have written more about this kind of constraint in our infrastructure audit readiness work.

# Step 1. Snapshot the current webhook config so we can prove what we changed.
kubectl get validatingwebhookconfiguration configmap-validator \
  -o yaml > /tmp/vwc-before.yaml

# Step 2. Flip failurePolicy to Ignore, scoped to this single webhook entry.
kubectl patch validatingwebhookconfiguration configmap-validator \
  --type='json' \
  -p='[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]'

# Step 3. Confirm the change before touching any ConfigMap.
kubectl get validatingwebhookconfiguration configmap-validator \
  -o jsonpath='{.webhooks[0].failurePolicy}'
# expect: Ignore

The unblock. Three commands, one of which is a snapshot for the post-incident review.

We patched the ConfigMap and the service was still broken

The env var that made the credential fix invisible

With failurePolicy on Ignore, the credential patch went through. We pulled the rotated values from the backup ConfigMap, applied them to the live one, and watched the profile service pods restart. /health still returned 500. The MongoDB connection error in the application logs still showed the old username.

That was the second moment in the incident where the model of the world had to change. The ConfigMap held the new credentials. The pod environment did not. Something else was setting MONGODB_URI on the container at runtime, and it was winning.

$ kubectl -n app get deploy profile-service \
    -o jsonpath='{.spec.template.spec.containers[0].env}' | jq
[
  { "name": "MONGODB_URI",
    "valueFrom": {
      "configMapKeyRef": {
        "name": "profile-mongodb-config",
        "key": "uri"
      }
    }
  },
  { "name": "PROFILE_MONGODB_URI_OVERRIDE",
    "value": "mongodb://oldapp:oldpw@mongo-old.app.svc:27017/profiles"
  }
]

The override. Set during a migration test six weeks earlier and never removed.

The application code read PROFILE_MONGODB_URI_OVERRIDE if it was set and otherwise read MONGODB_URI. The override had been added during a migration drill six weeks ago, never cleaned up, and was now silently shadowing every ConfigMap update we tried to apply. We have stopped accepting break-glass env overrides on production Deployments for this exact reason. If the override is worth setting, it is worth its own ConfigMap with an expiry annotation that a controller cleans up. Naked env values on the Deployment spec are invisible to the operators who do not know to look for them.

We removed the env var, the pod rolled, and /health came back as 200 on the third pod we curled.

Putting the webhook back, and making sure this never happens the same way again

Restoring failurePolicy: Fail without re-creating the trap

Before we restored failurePolicy to Fail, we fixed the webhook pod. The liveness probe path went from /healthz to /health, the value we had read from the compliance ConfigMap. The pod came up healthy and stayed up. We confirmed the webhook was actually answering by sending a deliberately invalid ConfigMap and watching the validation rejection come back cleanly. Only then did we flip failurePolicy back.

The harder problem was structural. A failurePolicy: Fail webhook that gates ConfigMap writes in a namespace is fine. A failurePolicy: Fail webhook that gates ConfigMap writes in a namespace that also contains the webhook itself, where the webhook's own Deployment depends on ConfigMaps in that namespace, is a bootstrap hazard. The first time something goes wrong, you cannot fix it without breaking your own policy.

The deadlock in one picture. Every fix path routes through a ConfigMap write the webhook itself is blocking.

We made two changes before we left. First, we moved the webhook's own Deployment, Service, and startup ConfigMap out of the app namespace and into a dedicated webhooks namespace, then set a namespaceSelector that excludes webhooks from validation, so the webhook can be rebuilt from its own in-cluster ConfigMaps even when it is the thing that is broken. Every ConfigMap in app is still validated, which is the whole point of the policy. Second, we added an objectSelector that excludes ConfigMaps carrying a specific break-glass label, so an on-call engineer with the right RBAC can apply a labelled emergency patch without flipping failurePolicy at all. Both changes were reviewed by the compliance team before we merged them, because relaxing the scope of a Fail-policy webhook is itself an audit decision.

The recovery script we left behind reads every value it needs (the health path, the ack token, the affected service list) from cluster state rather than hardcoding. Hardcoded recovery scripts go stale within a quarter; scripts that read from a compliance-owned ConfigMap stay correct as long as the source of truth is maintained. The script is idempotent: rerunning it on an already-recovered cluster is a no-op, which matters because the on-call engineer who runs it at 3 am should not have to think about whether they are the first or the third person to run it that night.

When to call us, and what we will look at first

If your admission layer is gating its own recovery

The thing that makes this incident shape hard is not the webhook itself. It is that the recovery path is non-obvious, the audit consequences of the obvious workaround (flipping policy or deleting the webhook config) are real, and the second-order trap (an env var on a Deployment shadowing the ConfigMap you just fixed) only shows up after you have already burned the credibility from the first workaround. Teams who hit this for the first time usually solve the immediate outage but leave the structural deadlock in place, and then it happens again on a different webhook six months later.

We run these recovery engagements every week. The admission-webhook-blocks-its-own-config shape has come up four times this year for us, once with cert-manager involved, twice with policy webhooks like this one, once with a service mesh sidecar injector that depended on a ConfigMap in its own namespace. The env-override-shadowing-a-ConfigMap-fix pattern is even more common; we see some version of it in roughly half of the credential rotation incidents we are called into.

If your cluster has a Fail-policy admission webhook today and you have never tested what happens when its pod is down, book an infrastructure review with our team and we will start with a 30-minute diagnostic call this week. We will walk your webhook configurations, identify the ones that gate their own dependencies, and give you a concrete plan to break the loop before an incident finds it for you.

Originally published at https://infraforge.agency/insights/admission-webhook-configmap-deadlock-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• A low-priority alert on log volume spikes that nobody investigated for days
• ConfigMap keys with URL values that contain '../' segments
• Application logs containing base64 strings that decode to credential-shaped prefixes
• ArgoCD Application source.repoURL values that point outside the expected repo root
• ConfigMap changes in the cluster that have no matching Git commit

Why a fanout service was emitting Keycloak credential fragments

The log line that should not have existed

An on-call engineer was triaging an unrelated paging storm and, out of habit, ran a Loki query against the noisiest service of the previous week. The fanout service had spiked from roughly 200 log lines per minute to 6,400 per minute three days earlier and had stayed there. The lines looked like garbage. They were not garbage.

{app="fanout-service"} |= "" | line_format "{{.message}}" | json | __error__=""

# Sample line (sanitized):
level=info msg="resolved source repo" repo="a2V5Y2xvYWstY2xpZW50LXNlY3JldDovL2NsaWVudC1pZD1ibGVhdGVyLWFwaQ==" component=helm-renderer

The base64 in the repo field decodes to 'keycloak-client-secret://client-id=bleater-api'. The fanout service was logging the resolved value of a config key that should never have resolved to a Secret.

We pulled the live ConfigMap. The offending key was named ARGOCD_APP_SOURCE_REPO_URL and its value was 'gitea.internal/platform/fanout/../keycloak-secrets'. That single '../' segment is the entire CVE-2022-24348 exposure. The ArgoCD Helm renderer, in vulnerable versions, would normalize the path after resolving it, walk out of the intended repo root, and read whatever Helm values or Secret references it found in the sibling directory. In this case the sibling directory was a Helm chart that templated the Keycloak client-secret Secret into its values. The fanout service's own application code, which logged its resolved configuration on startup and on every reconcile, then dumped fragments of that Secret into Loki as base64.

Three days. The fanout service itself was healthy the entire time. RabbitMQ consumers were running, distribution was working, the SLO board was green. The exposure was completely silent from a functional standpoint.

Log volume alerts without log content are noise generators

Why the alert sat for three days

The Grafana OnCall alert that fired three days earlier said, in effect, 'fanout-service log volume is 30x baseline'. It was tagged P3 and routed to a Slack channel that the team treats as a digest. The runbook attached to the alert said to check for retry loops. The on-call engineer at the time did check, saw no retries in the RabbitMQ metrics, and silenced the alert for 24 hours. The silence got renewed twice by the rotation handoff.

This is the part of the story we keep seeing across client engagements. A log volume alert that does not inspect log content tells you something changed, not what changed. If the alert had matched on the byte pattern of base64 strings longer than 32 characters in a service that does not normally emit base64, the page would have been P1 and would have gone to a human within minutes. Volume alone is not a signal anyone can act on in under an hour, so it gets silenced.

We have written more on this in our GitOps and ArgoCD recovery cluster, where the same pattern shows up under different vectors. The constant is that GitOps systems concentrate trust in the manifest pipeline, and any leak in that pipeline tends to surface first as 'weird logs' before it surfaces as anything else.

The pod restart that the ConfigMap patch alone does not give you

Patching the ConfigMap was not the fix

The instinct, once we identified the bad key, was to kubectl edit configmap and delete the line. We did not do that, for two reasons. First, the ConfigMap was managed by ArgoCD; a live edit would last until the next sync. Second, even after the ConfigMap was clean, the existing pods would still have the malicious URL in their environment because envFrom only resolves at pod start. The leak would continue until the pods were rolled.

The correct sequence had four steps and the order mattered.

# Verify the ConfigMap is clean
kubectl get configmap fanout-service-config -n bleater -o json \
  | jq '.data | keys[]' | grep -i repo_url
# (should return nothing)

# Verify pods restarted after the ConfigMap fix timestamp
kubectl get pods -n bleater -l app=fanout-service \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.startTime}{"\n"}{end}'

# Verify the ArgoCD Application source has no traversal
kubectl get application fanout-service -n argocd \
  -o jsonpath='{.spec.source.repoURL}' | grep -F '../' \
  && echo 'STILL VULNERABLE' || echo 'clean'

# Loki check, post-restart window only
logcli query --since=15m \
  '{app="fanout-service"} |~ "a2V5Y2xvYWs|keycloak"'

The four checks we ran in order. Any non-empty result on any of them would have meant the remediation was not complete.

Treating the leak as breached until proven otherwise

Auditing whether the Secret was actually read

The harder question was not 'did we stop the leak' but 'did anyone read the leaked data while it was leaking'. The fragments were in Loki, which meant anyone with Loki read access to the bleater namespace logs could have seen them. We pulled the Loki audit log for the three-day window and listed every query that matched fanout-service logs. Twelve queries from four engineers, all of them looking at unrelated debugging work, none of them filtering on the byte pattern that would have exposed the credential.

That was reassuring but not sufficient. The Keycloak client-secret had to be treated as compromised regardless, because we could not prove the absence of external log exfiltration with high confidence. We rotated the client-secret, redeployed the services that used it, and audited Keycloak's own access log for any token issuance using the old secret from an unexpected source IP in the exposure window. We found none. The rotation took about 25 minutes including service redeployment.

We then went back to the original question that nobody had asked yet: how did the malicious ConfigMap key get there in the first place. The automation script that applied it was a 'config sync' job that pulled key-value pairs from a shared spreadsheet and wrote them into the ConfigMap. The spreadsheet was editable by a wider group than the cluster was. Somebody had added the URL three days earlier, probably as a copy-paste mistake from a different document, and the sync job had faithfully applied it. There was no Git commit, no PR, no review.

Three controls that close this class of failure

What we changed so it cannot happen quietly again

We made three changes after this incident, in priority order.

The first was an admission webhook that rejects any ConfigMap apply where a string value contains '../' or matches the shape of a URL pointing outside an allowlist of internal domains. The rule is 12 lines of OPA policy. We tested it against six months of historical ConfigMap diffs and it would have caught this exact incident on day zero. It also catches the more common case of someone pasting a localhost URL into a shared config.

The second was retiring the spreadsheet-driven sync job. Every ConfigMap that lands in the cluster now comes from a Git commit, has a commit SHA annotation, and fails admission if the annotation is missing or does not match a real commit in the repo. The work to migrate the existing key-value pairs took about a week. The job is gone and is not coming back.

The third was rewriting the log volume alert. The new version fires when fanout-service log lines contain base64-encoded strings longer than 24 characters at a rate above 5 per minute, scoped to services that do not normally emit base64. It is a Loki recording rule with a regex match and it pages a human at P1. The first week it ran it caught two false positives (both were legitimate JWT logging that we then removed) and zero real incidents. We consider that a healthy signal-to-noise ratio for a security alert.

We also upgraded ArgoCD past the CVE-2022-24348 fix line. That should have happened a year earlier. If you are running an ArgoCD version older than 2.3.0, 2.2.9, or 2.1.15, stop reading this and go check, because the same vector is sitting in your cluster waiting for an unlucky ConfigMap edit.

The control surface after the incident. The two new gates are the admission webhook on apply and the content-aware log alert at runtime.

When a CVE has been silently active in your cluster for days

If you are staring at a similar exposure window

The hard part of this kind of incident is not the patch. The patch is one line. The hard part is reconstructing the exposure window with enough confidence to know what to rotate, what to disclose, and what to audit. That work requires log retention you can query precisely, audit trails for the systems that read those logs, and the discipline to treat any leaked credential as compromised until the access logs say otherwise. Teams that have not rehearsed this work tend to do all three poorly the first time.

We run GitOps and ArgoCD recovery engagements every month. We have seen the path traversal CVE three times in the last year, all on clusters running ArgoCD versions the operators thought were current, and we have seen the same 'silent ConfigMap injection via shared spreadsheet' antipattern more often than that. The remediation pattern is the same. The audit pattern is the same. The controls that close the gap are the same.

If you suspect a similar exposure in your cluster right now, request an infrastructure review and we will start with a 30-minute diagnostic call this week to scope the audit window and the rotation list. If the exposure is active, we will be on a bridge with your team the same day.

Originally published at https://infraforge.agency/insights/argocd-cve-2022-24348-path-traversal-secret-leak-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• OnCall UI Acknowledge and Resolve buttons spin and time out with a generic 500
• New alerts from real degradations get deduplicated into an incident that cannot be cleared
• OnCall pod logs show ORM errors referencing a column that does not exist in the table
• The Helm post-upgrade migration job reported success but Postgres logs show a lock_timeout on one ALTER TABLE
• There is no Prometheus alert on OnCall's own API error rate, so the regression went undetected

72 hours of swallowed alerts and two zombie incidents absorbing all of them

The alerting platform was the incident

When we got on the bridge, OnCall's incident list looked almost healthy. Two incidents in firing state, both from three days earlier, both with zero acknowledgment events. That should have been impossible. The on-call rotation had been live the whole time, and the runbook said any firing incident over 15 minutes old gets escalated. Nothing had been escalated because nothing new had appeared. Every alert fired by Prometheus Alertmanager in those 72 hours had been deduplicated by labels and folded into one of those two zombies.

The first thing we tried was the obvious one. Click Acknowledge in the UI. The spinner ran for about 20 seconds and the page returned a 500. Same for Resolve. Same for Snooze. Same when we called the API directly with curl. The web pods were up, the database was reachable, Redis was fine. Nothing in any dashboard suggested a problem, because nobody had built a dashboard that watched OnCall itself.

$ curl -s -X POST -H "Authorization: Bearer $TOKEN" \
    https://oncall.internal/api/v1/alert_groups/I8KZ.../acknowledge/
{"detail": "Internal server error"}

# from the oncall-engine pod
$ kubectl logs deploy/oncall-engine -c engine --tail=50 | grep -A2 ERROR
DatabaseError: column alerts_alertgroup.acknowledged_by_confirmation_phone does not exist
LINE 1: ...ledged_by_user_id", "alerts_alertgroup"."acknowledged_by_co...

The ORM was reaching for a column the table did not have.

A silent ALTER TABLE timeout the Helm hook never noticed

Why the migration job exited 0 with a half-finished schema

Our first guess was a bad release. The previous Helm upgrade had bumped OnCall by a minor version, and we assumed the new application code was looking at a field that genuinely had not shipped yet. That was wrong. The release notes said the column had been added in this version, and django_migrations on the OnCall database said the migration had been applied. Both things were true, and the column was still not there.

The clue was in Postgres logs from three days earlier, exactly when the Helm post-upgrade hook ran the migration job. One line, easy to miss, in the middle of dozens of normal statement logs:

2026-05-11 02:14:07 UTC ERROR:  canceling statement due to lock timeout
2026-05-11 02:14:07 UTC STATEMENT:  ALTER TABLE alerts_alertgroup
    ADD COLUMN acknowledged_by_confirmation_phone varchar(20) NULL;
2026-05-11 02:14:07 UTC LOG:  duration: 30001.114 ms

alerts_alertgroup is one of the highest-write tables in OnCall. At 02:14 a backlog of inserts was holding row locks, the ALTER hit the lock_timeout we had set globally to 30 seconds (a sensible default we put in years ago to stop one bad migration from wedging the whole database), and Postgres killed the statement. The migration script caught the exception, logged it to stderr, moved on to the next statement, and finished. The Helm hook checked the job's exit code, saw 0, and marked the release Succeeded. ArgoCD synced. The new pods rolled. And from that moment, every code path that touched the new column returned 500.

A plain retry would not have worked either, for a second and separate reason. The failed attempt had left a pgtrigger object behind on alerts_alertgroup, which we only found by querying pg_trigger directly, so Django's migration would have blown up on replay trying to create a trigger that already existed. That leftover trigger did not cause the lock_timeout (an existing trigger does not block an ALTER TABLE; only conflicting locks held by other sessions do). It just meant the migration could not be re-run cleanly until we removed it, and the ALTER itself still needed a window where the write backlog was not holding the table.

Why we forward-fixed instead of rolling the Helm release back

Drop the trigger, add the column, then unstick the zombies

We considered rolling back to the previous OnCall version. It looked clean on paper: the old image did not need the missing column, so the schema would match again and acks would work. We talked ourselves out of it for two reasons. First, the new pods had been running for three days and had written data shaped for the new version, including new fields in adjacent tables. A rollback would have meant either accepting writes that the old code did not understand or restoring a 72-hour-old database snapshot, which would erase three days of incident history including the zombies we wanted to clean up. Second, the next upgrade would just hit the same lock_timeout the same way. We would be back here in a week.

Forward-fix it was. The sequence had to be careful, because the table was still taking writes and we were going to ALTER it. We picked a low-write window, paused Celery workers that wrote to alerts_alertgroup (not the web tier, which we wanted up so the API stayed responsive), and ran the work inside one transaction:

-- 1. confirm the column is genuinely missing
SELECT column_name FROM information_schema.columns
WHERE table_name = 'alerts_alertgroup'
  AND column_name = 'acknowledged_by_confirmation_phone';
-- (0 rows)

-- 2. find the blocking trigger left over from the failed attempt
SELECT tgname FROM pg_trigger
WHERE tgrelid = 'alerts_alertgroup'::regclass
  AND tgname LIKE 'pgtrigger_%';

-- 3. drop it inside the same transaction we ALTER in
BEGIN;
SET LOCAL lock_timeout = '5min';
DROP TRIGGER IF EXISTS pgtrigger_oncall_protect_finished
  ON alerts_alertgroup;
ALTER TABLE alerts_alertgroup
  ADD COLUMN acknowledged_by_confirmation_phone varchar(20) NULL;
COMMIT;

Raise lock_timeout for this transaction only; do not touch the global.

We did not change the global lock_timeout. Setting it LOCAL inside the transaction lets this one ALTER wait up to five minutes, and any other migration that runs in normal conditions still gets the 30-second guard. Once the column existed, we unpaused the Celery workers and watched the engine pod logs. The 500s stopped within seconds.

That left the zombies. Acknowledging them was not enough. An acknowledged incident still sits in the firing state from OnCall's deduplication perspective, so new alerts would still fold into it. We had to mark them resolved. We did it through the API first to make sure the lifecycle hooks fired and downstream integrations got the resolved webhook. That worked for one of the two. The API still refused the other for an unrelated reason: its integration had been deleted, so OnCall could not look up the routing to fire the webhook. For that single record we set resolved=TRUE and resolved_at to the current timestamp in the database directly, with a note in the incident's raw payload explaining the manual close.

We then fired a synthetic alert from Alertmanager and watched a new incident appear, ack it from the UI in under two seconds, resolve it, and confirm a follow-up alert created a fresh incident instead of folding into the resolved one. That was the real all-clear.

Meta-monitoring for the platform that does the monitoring

What we wired up so the next silent migration trips an alarm

The thing that kept us up afterward was not the migration. Migrations fail. Database locks happen. The thing that kept us up was that OnCall had been broken for three days and not one signal in our monitoring stack had told us. We had alerts on Prometheus being down, on Alertmanager being down, on Grafana being down, on every customer-facing service. We had nothing watching the incident management platform itself.

We added two rules the same week. The first is a straight error-rate alert on OnCall's API. If more than 1% of requests to /api/v1/ return 5xx for five minutes, page the platform team at critical severity. Five minutes is short enough that a real outage gets caught but long enough that a single bad deploy rolling does not page. We picked critical because if OnCall is degraded, nothing else paging matters; alerts get swallowed.

groups:
- name: oncall-meta
  rules:
  - alert: OncallApiErrorRateHigh
    expr: |
      sum(rate(django_http_responses_total_by_status_total{job="oncall",status=~"5.."}[5m]))
      /
      sum(rate(django_http_responses_total_by_status_total{job="oncall"}[5m]))
      > 0.01
    for: 5m
    labels:
      severity: critical
      service: oncall
    annotations:
      summary: "OnCall API returning >1% 5xx for 5m"
      runbook: "https://internal/runbooks/oncall-api-errors"

  - alert: OncallMigrationJobStderr
    expr: |
      sum(increase(kube_job_status_failed{namespace="oncall"}[10m])) > 0
      or
      sum(increase(log_messages_total{namespace="oncall",app="migration",level="ERROR"}[10m])) > 0
    for: 1m
    labels:
      severity: critical
      service: oncall

The second rule catches a migration job that logs errors even if its exit code is 0.

The second rule is the lesson from this specific incident. Helm trusts the exit code. Django migrations swallow individual statement errors and continue. The only place the truth lives is in the job's log stream. We now alert on ERROR-level log lines from any pod with the migration label in the oncall namespace, regardless of whether the job reported success. We have caught two real issues with this rule in the months since (neither as bad as this one, both worth knowing about within minutes instead of days).

The broader pattern, and one we now apply on every recovery engagement we run, is that any tool you depend on to notice problems needs an independent way to notice when that tool itself is the problem. We have written more about this category of failure in our migration recovery work, because the same shape appears in database cutovers, queue platform upgrades, and identity provider migrations: the system you rely on to tell you the truth is the system that has stopped telling the truth, and you only find out from a customer.

When acks are silently 500ing and you cannot tell what data is real

If your OnCall is doing this right now

The hard part of this incident is not the SQL. The hard part is making the call between forward-fix and rollback when your incident history, your zombie state, and your live alert routing are all entangled in a database that is currently being written to by application code that expects a schema it does not have. Roll back without a plan and you lose three days of incident records. Forward-fix without checking for leftover triggers and migration locks and your second attempt fails the same way as the first. Run an ALTER on a hot table during business hours and you find out what your application's actual timeout tolerance is.

We do these engagements every few weeks. Partial Django migrations on Grafana OnCall is the specific case we have now seen three times this year, twice from lock_timeout and once from a leftover pgtrigger object that made the migration fail on replay. Adjacent variants we have handled: Sentry post-deploy migrations that left a column nullable when the code expected NOT NULL, Mattermost upgrades where one index creation timed out, Keycloak realm migrations that completed on the primary but failed on a replica. The pattern is identical and the recovery sequence rhymes.

If your team is staring at a 500 on every ack and trying to decide whether to roll back the Helm release, book an infrastructure review with our team and we will be on a bridge with you the same day. We will help you confirm the schema delta, plan the forward-fix or the rollback with the data implications spelled out, and clean up the zombie incidents without losing the history you need for the postmortem.

Originally published at https://infraforge.agency/insights/grafana-oncall-stuck-incidents-partial-migration/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• EBS Snapshot line is several hundred dollars a month and no active EBS snapshot pipeline is running on the account
• describe-snapshots --owner-ids self returns thousands of entries when you expect dozens
• Sampling a few snapshot IDs shows SourceVolumeId values that no longer resolve in describe-volumes
• A backup Lambda or custom snapshot script was deprecated in the last 12 to 24 months
• AWS Backup is the active tool and its dashboard shows normal counts, but the cost line tells a different story

$1,830 a month on a backup product the account no longer used

The line item that should have been zero

The EBS Snapshot line had been climbing slowly for thirteen months. Nobody had flagged it. The quarterly cost review surfaced it because the line item ranked sixth on the account, and the team's mental model said it should have ranked nowhere. There was no EBS snapshot policy running. AWS Backup had taken over RDS and EBS backups a year earlier, with the old Lambda plus EventBridge pipeline retired the same week.

The first instinct in the room was to pull AWS Backup's plan and see if a retention window had widened. The plan was clean. Snapshot counts there were in the low dozens, exactly what the new policy specified. So the snapshots driving the bill were coming from somewhere else.

$ aws ec2 describe-snapshots --owner-ids self \
    --query 'length(Snapshots)' --output text
9408

The number that turned a routine cost review into an incident.

That number was the moment the room got quiet. AWS Backup writes maybe forty snapshots a month on this account. Nine thousand was a different category of problem.

AWS Backup was clean, so who made these 9,408 snapshots

Ruling out the obvious suspect

With AWS Backup ruled out and no other named pipeline running, the question became: who created these 9,408 snapshots, and is anything still creating more. We pulled the StartTime field on the most recent hundred. The newest one was thirteen months old. Whatever pipeline made them had stopped, which meant we were looking at a stable population, not a leak that was still growing. That mattered because it meant the cleanup had a known size.

The next question was whether the source volumes were still around. We sampled twenty random snapshots and ran describe-volumes against their SourceVolumeId. All twenty came back InvalidVolume.NotFound. The pattern was clear: the snapshots were referencing two specific volume IDs (the Lambda snapshotted two production EBS volumes every two hours), both of which had been deleted along with the EC2 instances they served when the application moved to a managed service.

aws ec2 describe-snapshots --owner-ids self \
    --query 'Snapshots[*].[SnapshotId,VolumeId,StartTime]' \
    --output text > all-snapshots.tsv

awk -F'\t' '{print $2}' all-snapshots.tsv | sort -u \
  | while read vid; do
      if ! aws ec2 describe-volumes --volume-ids "$vid" \
          >/dev/null 2>&1; then
        echo "$vid orphan"
      fi
    done > orphan-source-volumes.txt

Group snapshots by their source volume, then check which source volumes still exist.

Only two volume IDs appeared in the orphan list. Two volumes, one snapshot every two hours, 392 days of runtime before the Lambda was deleted: 2 x 12 x 392 is 9,408. The arithmetic closed exactly, which told us we were looking at the whole population and not a subset of it. The Lambda that created them was gone, but AWS does not garbage-collect snapshots when their creator disappears. Snapshots are first-class objects with their own lifecycle, and that lifecycle is whatever you set when you create them. The Lambda set nothing.

Why we sampled twenty before touching the other 9,388

What we did before running delete-snapshot in a loop

The temptation at this point is to write a one-line loop and delete everything. delete-snapshot is irreversible. The cost was real, $1,830 a month for storage of data that referenced infrastructure that no longer existed. Two reasons we did not run the loop immediately.

First, orphan is sometimes a transient state. A volume gets deleted on Tuesday during a planned migration. On Wednesday the orphan-finder runs. A snapshot taken two hours before the volume's deletion looks orphaned but is actually the most recent backup of a service that was just migrated. Deleting it would destroy the only remaining copy of that data. We checked the StartTime on every snapshot in our sample against the deletion date of its source volume. Every one was older than the deletion by at least nine months. The cohort was uniformly historical. No active workflow could be depending on any of them.

Second, we needed to be sure these snapshots were not being referenced as the base for any AMI or any live AWS Backup recovery point. We ran describe-images with a block-device-mapping.snapshot-id filter on the sample, expecting nothing, and got nothing. We checked the AWS Backup recovery point inventory. None of the orphan snapshot IDs appeared there. The deletion was safe.

The delete loop was about forty minutes of actual work, and we spread it across three calendar days on purpose. delete-snapshot is rate-limited at roughly 5 requests per second per account with bursts, so 9,400 deletes plus retries on the occasional 503 is well under an hour of throughput even at a conservative pace. Nobody wanted nine thousand irreversible deletes running in one unsupervised burst, so we split the list into three batches of about 3,100, one per day, each batch gated on someone reading the previous day's failed.log and signing off before the next one started. The loop itself had a 250ms sleep, a checkpoint file, and an append-only deleted.log so we could resume after any interruption without re-trying ones that already succeeded.

while read sid; do
  if grep -qx "$sid" deleted.log; then continue; fi
  aws ec2 delete-snapshot --snapshot-id "$sid" \
    && echo "$sid" >> deleted.log \
    || echo "$sid" >> failed.log
  sleep 0.25
done < orphan-snapshot-ids.txt

Resumable, rate-limited delete loop. The checkpoint file is the load-bearing part.

After three days the EBS Snapshot line on the next monthly forecast dropped to under $20. The thirty-six terabytes of orphan storage was gone.

Tag at creation, schedule the cleanup, watch the lines that should be zero

The rule that meant the next deprecated pipeline could not do this

The deletion fixed the symptom. The interesting part of this engagement was the cause. AWS does not couple a snapshot's lifecycle to the lifecycle of whatever process created it. A Lambda gets deleted, an EventBridge rule gets removed, the IAM role goes with them, and the snapshots they made keep existing and keep being billed, forever, until something explicitly deletes them. There is no warning email. There is no dashboard widget. The only signal is the monthly bill, and the bill takes a year to be loud enough to investigate.

Two changes went in after the cleanup. The first was a tag-at-creation rule. Every snapshot the account creates now carries three tags applied at creation time: Owner (a team or service name), Retention (an ISO date past which the snapshot is safe to delete), and CreatedBy (the pipeline that made it). AWS Backup applies these automatically through its backup plan. The handful of custom Lambdas that survived the migration were rewritten to apply them. A weekly cleanup Lambda walks the account, deletes anything past its Retention date, and flags anything older than 90 days with no Retention tag. For the first 60 days the Lambda posted a Slack message and waited for a thumbs-up before deleting. After that it ran unattended.

The second change was to the quarterly cost review process. It now starts with the line items that should be zero or near zero, not the ones that are already big. The big lines get watched constantly by capacity planners. The lines that should be zero are where deleted infrastructure leaves footprints, and they are the ones least likely to be on anybody's dashboard. EBS Snapshot on a no-EBS-snapshot account. Lambda invocations on a service that was migrated to ECS six months ago. NAT Gateway hours on a workload that should not need cross-AZ egress. These are the lines where deprecated pipelines keep paying rent.

The lifecycle every snapshot now goes through. Untagged snapshots cannot live past 90 days without an explicit decision.

Cost archaeology on accounts where a deprecated pipeline is still paying rent

When the bill is the only thing telling you what you forgot

The shape of this incident is common. A pipeline gets shipped, the engineer who wrote it leaves, the policy gets replaced but the outputs survive, and the bill slowly bends upward. EBS snapshots are the most common shape we see. Detached EIPs are close behind. Idle NAT gateways and orphaned ElastiCache clusters round out the top four. None of these line items alarm on a CloudWatch dashboard because nothing is actively misbehaving. The deprecated pipeline is the misbehavior, and the pipeline no longer exists.

We run these cost-archaeology engagements regularly. In the last quarter we walked through three accounts where a single deprecated backup pipeline accounted for more than half of the account's EBS Snapshot line. We have an inventory script that finds orphan snapshots, detached volumes, unused EIPs, and idle NAT gateways across an account in about 20 minutes, plus a sample-then-delete workflow we walk the team through live so nothing irreversible happens on autopilot. The deletion is always the easy half. The work is figuring out which orphans are safe and writing the tag-at-creation policy that stops the next one.

If your bill has a line that does not match anything that should be running, the orphan audit is usually the fastest way to find out where it is going. Request an infrastructure review and we will run the audit with your team on a 30-minute diagnostic call this week. You can also see the broader pattern in our services overview for cloud cost spike work.

Originally published at https://infraforge.agency/insights/orphan-ebs-snapshots-deleted-backup-pipeline-cost-spike/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• Every PR touching a shared module shows N service plans in CI, each with 50+ resource changes
• Reviewers approve with 'plans look normal' without scrolling through them
• A single shared module has accumulated 25 to 40 input variables to handle per-service edge cases
• CI plan time grows linearly with consumer count: one module change forces all N consumer plans to run, each loading its own remote state
• A bug in the module breaks multiple unrelated services in the same apply window

How a 1,400-resource plan output stopped being read

The PR that broke three services had a clean LGTM

The original consolidation was, on paper, exactly the refactor every platform team is told to do. Fourteen service-specific Terraform configs, each maintained by a different feature team, each with its own subtle drift from the others. The platform team pulled the common shape out into one service-stack module, parameterized the differences, and pointed all 14 services at it. Eight thousand lines of HCL became twelve hundred. A change to add a shared observability sidecar landed across all 14 services in a single PR. Everyone celebrated.

The failure mode took six months to surface because the early signals looked like wins. Module changes shipped faster than the per-service changes they replaced. The platform team felt productive. What nobody tracked was that the CI plan output for every module PR had grown from one service's plan to fourteen, and the reviewers had silently adapted by reading the first plan, skimming the second, and rubber-stamping the rest.

Then a module-level change to how IAM policies were attached introduced a subtle bug: for services that overrode the default policy document, the new code path replaced rather than merged. Three of the 14 services overrode that default. The plan output showed the destruction and recreation of those policy attachments quite clearly, on lines somewhere around 870 of the GitHub diff view. The PR had three approvals.

Title: fix: typo in service-stack variable description

Diff: 1 line changed (a comment)

CI: terraform-plan-all ✓
  - service-a/plan: 84 changes
  - service-b/plan: 91 changes
  - service-c/plan: 102 changes
  - service-d/plan: 88 changes
  ... (10 more)

Total: 1,388 resource changes

Reviews:
  @platform-lead   approved 'LGTM, plans look normal'
  @service-b-eng   approved 'looks fine'
  @service-g-eng   approved

What the PR description looked like, paraphrased from the post-mortem

The fix that was not reviewer discipline

What we thought it was, what it actually was

The first instinct, and the one the team had spent two weeks pursuing before we got involved, was that this was a code review hygiene problem. They had written a PR template that required reviewers to acknowledge they had read each plan. They had a Slack bot that posted a daily "unreviewed plan changes" count. The platform lead had given a brown bag talk titled "Read Your Plans." None of it stuck, because none of it could stick. Asking a human to read 1,400 lines of plan output for a one-character comment fix is asking them to do something nobody should do, and they will not do it for long even if you make them feel guilty about it.

The actual problem was structural. The module had become a dependency surface that 14 consumers were forced to redeploy together, on every change, whether the change affected them or not. That is not a code review problem. That is the same coupling problem distributed systems people argue about with monoliths and microservices, except it had snuck in through the back door of a Terraform refactor. The cost of coupling does not show up the day you consolidate. It shows up the first time a small change has to ship and the blast radius is the entire fleet.

We have written more on the broader pattern in the Terraform and IaC debt pillar, but the specific recovery for this shape of problem has three layers, and they have to land in order.

How we cut the blast radius from 14 to 1 in an afternoon

Pinning the module per service was the bleeding stopper

The immediate move was to stop every consumer from being forced to re-plan on every module change. The mechanism is dumb and effective: pin each service's module reference to an explicit git ref instead of letting them all track main.

# Before: every consumer floats on main
module "service" {
  source = "git::https://github.com/org/modules.git//service-stack"
  name   = "auth-api"
  # ...
}

# After: every consumer is pinned to an explicit version
module "service" {
  source = "git::https://github.com/org/modules.git//service-stack?ref=v1.4.2"
  name   = "auth-api"
  # ...
}

Before and after: the module block in each service's Terraform config

After the pin, a module change ships as a tagged release in the modules repo, then ships to consumers one at a time via a per-service PR that bumps the ref. Each of those PRs shows exactly one service's plan, and that plan is short enough to read. The reviewers can do their job again. The author has to think about which services they actually want this change in, in what order, and on what schedule.

There is a real cost to this, and we want to name it honestly: you have given up some of the consolidation win. You can no longer ship an observability change to all 14 services in one PR. You can ship it in one tagged module release plus 14 small bump PRs, which is more clicks. We have not had a client regret the trade once they lived with it for a month. The clicks are cheap; the missed bug in plan number nine is not.

The change-propagation shape before and after pinning

Why the 30-input module became 5 inputs plus an advanced object

Tiering inputs and splitting by change velocity

Pinning bought time. It did not fix the underlying reason the module had become hard to change. We sat with the platform team and looked at all 30 inputs the module had grown. Most services used 5 of them. The other 25 existed because, over six months, individual services had asked for an escape hatch ("can the module take a custom IAM policy document?", "can we override the security group rules?", "can we set a node selector?") and the module owners had said yes, every time, because saying no felt like blocking a teammate. The module had become an everything-bagel.

We refactored the input surface into two tiers. The common five became first-class top-level inputs. The other 25 went into an optional advanced object with optional() fields, so a normal consumer never sees them and an exotic consumer has to opt in deliberately.

# Common path: every service uses these
variable "name"        { type = string }
variable "image"       { type = string }
variable "replicas"    { type = number }
variable "environment" { type = string }
variable "port"        { type = number }

# Escape hatch: explicit, optional, and visible in code review
variable "advanced" {
  type = object({
    custom_iam_policy_json = optional(string)
    extra_sg_rules         = optional(list(object({ ... })))
    node_selector          = optional(map(string))
    # ... 22 more rarely-used knobs
  })
  default = {}
}

variables.tf after the tiering refactor

Then we did the harder work: splitting the module along change-velocity boundaries. The monitoring submodule was changing roughly weekly, the database submodule once a quarter, and the networking submodule about twice a year. Bundling them together meant every monitoring tweak forced a re-plan of database and networking resources for all 14 consumers. We pulled them apart into separate modules with separate version pins, so a consumer can bump monitoring from v2.1 to v2.2 without touching database at all.

This is the same argument microservice advocates make about service boundaries, and the rule of thumb is the same: couple things that change together, decouple things that change at different rates. The cost of getting it wrong in Terraform is not latency or distributed-transaction pain. It is plan output nobody reads, and bugs that ship because of it.

The OPA gate is worth its own sentence. It is twenty lines of Rego that counts distinct workspaces touched by the plan and fails the PR over a threshold unless the author explicitly opts in. It does not prevent fleet-wide changes; it forces the author to acknowledge they are making one. That single check has caught two accidental fleet-wide PRs at the clients who have adopted it, both of which would have shipped under the old regime.

When the consolidation win has become a coupling tax

If your shared modules feel like this right now

The hard part of this kind of recovery is not the technical work. Pinning a module ref is ten minutes of typing per service. Splitting a module along change-velocity lines is a weekend. The hard part is convincing the team that the consolidation they are proud of has become a liability, and doing the unwind in an order that does not cause an outage. We have done this engagement at four SaaS platforms in the last year, and the pattern of "reviewer fatigue followed by a buried bug" shows up in three of the four. The fourth caught it before the bug shipped, only because their CI plan output had grown past GitHub's diff size limit and forced the conversation.

We run these recovery engagements every week. If your platform team is shipping module changes that produce thousand-line plan outputs and your reviewers have started writing "plans look normal" without reading them, the next bug is already on its way. Book an infrastructure review with our team and we will spend a 30-minute diagnostic call this week mapping your module consumer graph and naming the first three pins to put in place.

Originally published at https://infraforge.agency/insights/terraform-shared-module-coupling-fleet-wide-plans/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• AWS bill jumped 2-4x in under a week with no traffic or feature change
• Cost Explorer concentrates the spike on DataTransfer-Regional-Bytes and RDSInstance line items
• An RDS read replica sits in a different AZ than its primary and shows jagged ReplicaLag (spikes to 30s, drops to 0.5s, repeats)
• No application config or BI tool actually points at the replica's endpoint
• Recent schema or replication change on the primary that nobody coordinated with replica owners

Chasing the analytics query that did not exist

What we thought it was first

Almost every cost spike I have seen in the last three years gets blamed on analytics first. There is usually a junior data person, a notebook, a forgotten SELECT *, and a story everyone tells themselves. So we did the natural thing. We pulled the Athena query history for the previous ten days. Nothing unusual. We checked Redshift, which the team barely uses. Idle. We checked the data warehouse cluster's autoscaling history. Flat.

The clue was in Cost Explorer, but only when we grouped by usage type instead of by service. The RDS line item was up, sure, but the line item that had really moved was DataTransfer-Regional-Bytes. That is the meter for cross-AZ traffic inside a single region. Analytics queries do not typically light that meter up unless somebody has put a compute node in one AZ and the data in another, which would have been a much weirder problem.

Cross-AZ data transfer at that volume meant something was constantly shipping bytes between two availability zones. The shape of the bill said: find the thing that talks to itself across AZs at high frequency.

How we found the orphan replica

The diagnostic turn

We listed every RDS instance in the account and compared the AZ of each replica to its primary. One read replica was in us-east-1b while its primary was in us-east-1a. That alone is not a problem; cross-AZ replicas exist for legitimate HA reasons. What was odd was that this replica was tagged with nothing. No Owner. No Purpose. No Environment. Just the default Name tag, which read load-test-replica-temp.

# List replicas with their AZ and their primary's AZ
aws rds describe-db-instances \
  --query 'DBInstances[?ReadReplicaSourceDBInstanceIdentifier!=`null`].[DBInstanceIdentifier,AvailabilityZone,ReadReplicaSourceDBInstanceIdentifier,DBInstanceStatus]' \
  --output table

# Then for each primary, get its AZ
aws rds describe-db-instances \
  --db-instance-identifier <primary-id> \
  --query 'DBInstances[0].AvailabilityZone'

The two commands that surfaced the orphan in about 30 seconds.

The replica's CloudWatch ReplicaLag metric was the giveaway that this was not a healthy idle replica. It would spike to 30 seconds, drop to 0.5 seconds, spike again, every minute or so. That sawtooth pattern means the replication thread is failing and retrying. We pulled the replica's error log and found the same line repeating roughly every 50 milliseconds: error 1153, got a packet bigger than 'max_allowed_packet' bytes. Someone had raised max_allowed_packet on the primary six days earlier to land a batch of large BLOB writes, and the replica was still on the old parameter group. Its IO thread pulled the first oversized transaction, blew the packet limit, dropped the connection, reconnected, and asked for the same binlog position again.

Every reconnect re-shipped the same binlog chunk across the AZ boundary, and the replica never got past it. At 50ms intervals, 24 hours a day, for six days. That was the bill.

The five-minute check that prevents the worse outcome

What we did before deleting anything

The instinct, when you have found the thing burning money, is to kill it immediately. We did not. The worse outcome here is not 'replica costs another hour of cross-AZ transfer'. The worse outcome is 'replica gets deleted, a quarterly BI dashboard breaks on Friday, and finance is back in your inbox with a different question'.

So we did the cheap verification first. We grepped the application monorepo for the replica's endpoint hostname. Zero hits. We checked the BI tool's data sources (Metabase in this case). Nothing pointed at it. We checked the data team's Airflow DAGs. Clean. We checked Terraform state to see how it had been created. It was in a workspace tagged load-test that had not been touched in a month, and the engineer who created it had left the company three weeks earlier.

aws rds delete-db-instance \
  --db-instance-identifier load-test-replica-temp \
  --skip-final-snapshot

The actual delete, once we were confident nothing depended on the replica.

Tag hygiene, expiration sweeps, and an anomaly budget that would have caught this on day 2

What we changed afterwards

Forgotten resources are the largest single category of cloud waste I see in client accounts. Bigger than oversized instances. Bigger than reserved-instance gaps. The fix is mechanical. Every cost-generating resource needs three tags: Owner, Purpose, ExpiresAt. ExpiresAt is the one most teams skip and the one that does the work.

We deployed a small Lambda on a weekly schedule that walks RDS, EC2, ELB, ElastiCache, and OpenSearch, finds resources past their ExpiresAt date or missing tags entirely, and posts to a Slack channel pinging the Owner. The owner has two weeks to either re-tag with a new ExpiresAt or delete. Resources with no Owner go to the platform team's queue. The first sweep flagged 47 resources across the account. Six of them were costing real money.

The sweep logic. About 180 lines of Python in practice.

The second change was AWS Budgets with anomaly detection scoped per service. The team had a single account-wide budget set at \(5,000/month, which is useless for catching this kind of incident because the spike was concentrated in one service and the account total only crossed \)5,000 on day five. A per-service budget on RDS set at $4,000 with a 20% variance threshold would have fired on day 2. The alert that matters is the one that fires before you have spent the money, not after.

The third change was a process one. The original max_allowed_packet change had been an uncoordinated parameter-group tweak from a senior engineer who had not realized a replica existed. Parameter-group and schema changes now require a checklist that includes 'list all replicas of this primary and apply the same parameter change to each' as a pre-flight step. It is not glamorous. It would have prevented the entire incident.

Where cost spike triage gets stuck

If your AWS bill just jumped and you do not know why

The hard part of a cost spike is not finding the resource. It is being confident enough to delete it. Most teams we work with have at least one orphan RDS, ElastiCache, or NAT gateway they are afraid to touch because nobody remembers what depends on it. The triage takes a day; the courage to act takes a week of meetings. By then the bill has run another $2,000.

We run cost spike triage engagements every month. We have seen the orphan-replica case four times this year, the NAT-gateway-in-the-wrong-AZ case more often than that, and a half dozen variants of 'load test that never got cleaned up' across CloudWatch Logs, OpenSearch, and Aurora Serverless. The pattern is almost always the same: a resource that nobody owns, a tag policy that was never enforced, and a budget alert tuned too coarse to catch concentration in a single service. We have written more on the underlying patterns in the cloud cost spikes problem brief and across our services.

If your AWS bill jumped this month and you cannot point at the resource with confidence, book an infrastructure review with our team and we will start with a 30-minute diagnostic call this week. Cost stops accruing the day we find the orphan.

Originally published at https://infraforge.agency/insights/forgotten-rds-replica-cross-az-cost-spike/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• Pods stuck in Init:0/3 or Init:1/3 with no forward progress and no clear log story
• kubectl edit deployment changes revert within ten to fifteen seconds, every time
• Three init containers each failing in a different protocol layer (TCP dial timeout, NXDOMAIN, AMQP ACCESS_REFUSED)
• A topology or schema ConfigMap claims state that the live broker or database disagrees with
• No timeout inside the init container wait scripts and no progressDeadlineSeconds on the Deployment, so a transient failure wedges the rollout indefinitely

Two replicas wedged, seven changes queued, four failed patches

The 2 am page

When we joined the bridge, the on-call engineer had already burned forty minutes on what looked like a config drift bug. The fanout service in the platform namespace had two replicas, both stuck in Init:0/3. The init container chain had three steps (wait-for-redis, wait-for-mongodb, wait-for-rabbitmq) and the redis step was failing on a hardcoded IPv4 address that did not match the live Service. They patched the env var on the Deployment. The init container restarted. Ten seconds later the IP was back. They patched it again. Same thing.

Their working hypothesis was etcd corruption or a faulty kube-apiserver caching layer. We have seen both before, but neither matches the symptom shape here. Etcd corruption surfaces as 5xx responses to kubectl, not as silent successful PATCHes that revert. We needed to find what was doing the reverting before we wasted any more time on the symptoms.

Two wrong guesses before the real culprit became visible

What we thought it was first

The first guess was a GitOps controller with self-heal enabled. ArgoCD does this with syncPolicy.automated.selfHeal: true. Flux does this with its Kustomization controller. Both will revert a kubectl patch within seconds if the live spec drifts from the source of truth in git. We checked the cluster for both. No Argo Application referenced the fanout namespace. Flux was not installed at all.

The second guess was a mutating admission webhook. A custom webhook that rewrites init container specs at admission time could in theory produce this pattern, except admission webhooks fire on create and update, not on a ten-second timer. We ran kubectl get mutatingwebhookconfigurations and the output was empty. That ruled it out.

The reverting was not coming from inside the cluster. It had to be coming from the node itself. We SSHed to the node where one of the fanout pods was scheduled and went looking. Within two minutes we had it.

$ ssh node-01 'ps -ef | grep admission'
root  1842  ... /usr/bin/supervisord -c /etc/supervisor/conf.d/admission.conf
root  2104  ... /bin/bash /var/lib/apex/admission.sh

$ ssh node-01 'cat /etc/supervisor/conf.d/admission.conf'
[program:admission]
command=/var/lib/apex/admission.sh
autorestart=true
startsecs=5

A supervisord-managed script on the node was the reverter. autorestart=true meant killing it bought us at most a few seconds.

The stored ConfigMap was the source of truth, not the live Deployment

What was actually overwriting our patches

The script at /var/lib/apex/admission.sh ran every ten seconds. It read three fields (redis-host, mongodb-host, amqp-uri) from a ConfigMap called fanout-init-config and patched them straight into the init container env vars on the live Deployment. The ConfigMap was the source of truth. The Deployment was a downstream artifact. Patching the Deployment was about as durable as writing in pencil.

The reverting loop. Edit the ConfigMap, not the Deployment.

This pattern shows up in places where the original GitOps story had gaps and someone wrote a node-side enforcer as a stopgap. Then the team rotated, the wiki page got out of date, and the enforcer kept running. We have seen this exact shape three times in the last year. Twice with supervisord scripts. Once with a systemd timer. The fix is always the same: find the source of truth before patching anything, and if you cannot find it in under fifteen minutes, stop and look on the nodes.

What each failure actually told us, and the fourth fix that did not show in any log

Three init containers, three different protocols

Once we knew to edit the ConfigMap, we still had three concurrent faults to diagnose. Each init container was failing in a different layer of the network stack, and each one had its own diagnostic signature.

The redis init container was dialing 10.43.181.44 on port 6379 and getting i/o timeout after thirty seconds. We compared against the live Service and got back a different ClusterIP.

$ kubectl get svc redis -n platform -o jsonpath='{.spec.clusterIP}'
10.43.218.92

$ kubectl logs fanout-7d4b9c-xx -c wait-for-redis -n platform | tail -3
dial tcp 10.43.181.44:6379: i/o timeout
dial tcp 10.43.181.44:6379: i/o timeout
dial tcp 10.43.181.44:6379: i/o timeout

The hardcoded IP had no relationship to the live Service. ClusterIPs are not stable across Service recreation. Hardcoding one is a time bomb.

The mongodb init container was logging 'lookup mongo.platform.svc.cluster.local on 10.43.0.10:53: no such host'. The live Service was named mongodb, not mongo. One character off, NXDOMAIN. We caught it by running kubectl get svc -n platform and reading the actual Service name out loud. The hostname in the ConfigMap had been typed from memory by someone who remembered the team's old naming convention.

The rabbitmq init container was the most interesting of the three. The TCP connection succeeded. The AMQP frame negotiation succeeded. Authentication succeeded. The vhost open returned ACCESS_REFUSED. The URI was amqp://app:app@rabbitmq:5672/fanout-internal. We port-forwarded to the management API and listed valid vhosts.

$ kubectl port-forward -n platform svc/rabbitmq 15672:15672 &
$ curl -s -u app:app http://localhost:15672/api/vhosts | jq -r '.[].name'
/
/platform

# fanout-internal does not exist on this broker

The URI parsed cleanly and authenticated cleanly. The failure was at vhost open. Always enumerate vhosts before assuming auth or credentials.

There was a fourth fix that did not show up in any log. Nothing bounded how long an init container was allowed to wait. Kubernetes has no per-container deadline field: activeDeadlineSeconds exists on a Pod spec and a Job spec, never on a container, and putting it on a Deployment's Pod template is worse than useless because it kills healthy pods once they have been active that long. The bound has to come from the wait itself. We wrapped every init command in timeout 120 so a hung dial exits non-zero after two minutes, and set progressDeadlineSeconds: 600 on the Deployment. Now a transient DNS hiccup or broker restart fails the init step fast and lets the kubelet retry the Pod, and a rollout that never converges reports ProgressDeadlineExceeded instead of sitting silently wedged.

A second ConfigMap with the same shape, intentionally broken, was a load-bearing canary

The look-alike ConfigMap we almost broke

Before we patched fanout-init-config, we almost made one more mistake. There was a second ConfigMap in the same namespace called fanout-init-config-canary. Same shape, same broken-looking IP, same broken-looking AMQP URI. It was labeled role: protected and annotated with purpose: chaos-canary. A drift-detection job in the cluster read it every fifteen minutes to confirm its own detection logic still fired on broken inputs. If we had run a sed-style global replace across all matching ConfigMaps (which is exactly what a tired engineer at 3 am tends to do) we would have silenced the canary and the team would have learned about the next round of real drift only when a customer noticed.

When you patch infrastructure under pressure, target the named resource, not the pattern. Read the labels and annotations of every resource you are about to touch. A surprising number of clusters have load-bearing decoys you do not know about until you break them. We have written more on this in the Kubernetes and CI/CD stabilization pillar.

Source-of-truth guard, deadline defense, a validation Job, and convergence checks

What we changed afterwards

The fanout service was the visible failure, but the recovery exposed four underlying gaps in the team's release flow. We left four durable changes in place before disconnecting from the bridge.

The fanout-init-config ConfigMap is now committed in git and synced via a real GitOps controller, and the node-side admission script was rewritten to refuse to overwrite a Deployment if the ConfigMap's content hash does not match a known-good baseline annotation. The script can still enforce, but it cannot enforce a broken state.

Every init container in the platform namespace now wraps its wait in timeout 120, and every Deployment sets progressDeadlineSeconds: 600. The pair matters. The in-container timeout fails the individual step fast instead of blocking forever on a dead dependency; the Deployment-level progress deadline surfaces a rollout that is structurally wrong instead of leaving it parked in Init. activeDeadlineSeconds stays where the API actually accepts it, on the pre-deployment validation Job.

A pre-deployment validation Job runs as part of the release flow. It carries label validation: predeploy, restartPolicy: OnFailure, activeDeadlineSeconds: 120, and a validator that does three real checks: redis, mongodb, and rabbitmq Services each have non-empty Endpoints, AND the broker reports every binding the topology ConfigMap claims to have declared. Topology drift was the other half of this incident; the binding count had silently dropped from five to three after a partial migration three weeks earlier, and nobody had noticed because the topology-version annotation still said 5.

# Snippet from the topology-reconcile Job that fixed the broker drift
apiVersion: batch/v1
kind: Job
metadata:
  name: topology-reconcile-2026-05-15
  labels:
    validation: predeploy
spec:
  activeDeadlineSeconds: 120
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: reconcile
        image: rabbitmq:3.13-management
        command: ["/bin/bash", "-c"]
        args:
          - |
            set -euo pipefail
            EXPECTED=$(yq '.bindings | length' /config/topology.yaml)
            for b in $(yq -o=json '.bindings[]' /config/topology.yaml | jq -c .); do
              EX=$(echo $b | jq -r .exchange)
              QU=$(echo $b | jq -r .queue)
              RK=$(echo $b | jq -r ."routing-key")
              rabbitmqadmin declare binding source=$EX destination=$QU routing_key=$RK
            done
            ACTUAL=$(curl -s -u $USER:$PASS http://rabbitmq:15672/api/bindings | jq 'length')
            [ "$ACTUAL" -ge "$EXPECTED" ] || exit 1

Reconcile via Job, not via kubectl exec. The Job is observable, retryable, and leaves an audit record.

The team's rollback runbook now requires two consecutive green health observations twenty seconds apart before a rollout is declared finished. Single-shot green is not enough on a cluster that has a ten-second admission tick, because you can catch the Pod between reverts and declare victory ninety seconds before the next failure cascade. We learned to distrust single-shot green the hard way on a different engagement, and that is now the default in every recovery handover we ship.

If you are looking at a cluster where every patch reverts within seconds, do not patch faster. Stop patching and find what is doing the reverting. The fix itself is usually ten minutes once you know where the source of truth lives. Finding the source of truth is what takes the hour. If you want a second pair of eyes on a system that is in this state, request an infrastructure review and we will be on a bridge with you the same day.

Originally published at https://infraforge.agency/insights/init-container-cascade-reverting-patches/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• terraform plan succeeds locally but terraform apply fails on a specific environment
• The error is Inappropriate value for module argument or Inappropriate value for attribute deep inside a child module
• The traceback points at a merge() or lookup() call inside a module that has not been edited in months
• Your root module input list has crossed 20 variables and several are typed any or map(any)
• There is no CI job that runs terraform plan against every environment on every PR

Three hypotheses, three dead ends, thirty minutes left in the release window

What we ruled out in the first 18 minutes

The first thing the on-call lead suggested was state drift. Someone, somewhere, had terraform import-ed a resource by hand. We checked the audit log. No import events in the past 30 days. We checked the lock table in DynamoDB. The lock had been released cleanly by the previous successful apply at 2:11pm.

The second hypothesis was provider version drift. The team had recently bumped hashicorp/aws from 5.62 to 5.71 in versions.tf. A breaking change in a resource schema can absolutely produce a type error like this one if apply pulls a newer provider than plan resolved against. We pinned both runs to 5.71 explicitly, deleted .terraform/, re-ran init, then plan, then apply. Same error, same module, same line.

The third hypothesis was a stale workspace. terraform workspaces sometimes diverge from the configuration if workspace select was bypassed by an engineer who exported TF_WORKSPACE and forgot. We ran terraform workspace show and verified it matched the intended target. The plan output even confirmed the right resource addresses.

Three explanations, three dead ends, eighteen minutes burned. The release window was now thirty minutes wide and shrinking. The on-call lead asked whether we should just roll back the deploy and figure it out Monday. I asked one more question first.

The 15th map(any) input that had been silently incubating for three weeks

Where the collision actually lived

I asked the on-call lead to walk me through what had merged into the workspace in the past three weeks. There were six commits. Five were obvious changes (image tags, a new IAM policy, a security group port). The sixth was a feature flag, added as a 15th map(any) input on the root module by an engineer who had joined six weeks earlier.

That was the lead.

The root module had 28 input variables. 14 of them were any-typed or map(any) to absorb per-environment overrides accumulated over six years of feature additions. The new feature flag added a 15th map(any) input named feature_overrides. Its values flowed through a merge() chain down to the database child module, which did its own merge(local.legacy_db_flags, var.feature_overrides) inside modules/services/database/locals.tf.

The two maps had a key collision. Both contained a key named read_replica_routing. The new input's value was a string. The legacy local's value was a map(object({ host = string, weight = number })). merge() resolves collisions by taking the last argument's value, and var.feature_overrides is the last argument, so wherever it carried that key it won. It only carried it in staging: the key was set in envs/staging.tfvars and left out of every other environment's tfvars, which is why staging was the one place the wrong-typed value ever reached the module.

How map(any) defers type-checking past plan and surfaces it at apply

The collision had been latent for three weeks. plan succeeded because terraform's planner walked the call graph with both maps' element types collapsed to any. The merged value passed type-check as any, which type-checks against anything. apply, which actually constructs the resource, evaluated the merged value against the receiving attribute's concrete type signature and discovered the value was a string where an object was required.

That is the part that hurts. Terraform's any type defers all type-checking until apply. Every map(any) input on a root module is a future apply-time failure waiting on a contributor who does not know the implicit shape.

Three options, one open release window, two minutes to pick

What we did before running apply again

We had three options and one open release window. I walked the on-call lead through them on the bridge call.

Option three carried the day. The rename took four minutes. The validation block took six. apply succeeded at 5:14pm with sixteen minutes to spare on the release window. The release shipped on time.

The audit work behind option one (the one we did NOT take) is what stuck with me. The next morning, we grep-ed the entire terraform/ tree for read_replica_routing to map every consumer. Seven references across four modules. Three in modules/services/database/locals.tf itself. One in modules/monitoring/cloudwatch.tf. One in modules/services/cache/lookups.tf, which read the value to construct its own routing decision and would have broken silently if we had deleted the legacy key the night before. The remaining two were in a state-recovery helper module the team had forgotten existed. We had nearly fired the second shot of our own foot.

We left a tombstone comment on the legacy key and an open PR that would, the following week, replace its map(any) type with a proper object({ ... }) schema. That work landed five days later. The downstream consumers caught the change at plan time, and three of them needed minor patches before the type tightening could merge. None of those patches would have caught the original collision. They all caught real existing bugs the any type had been hiding.

Two policy changes and one structural fix

What we changed afterwards

Two policy changes came out of that night, and one structural fix took longer.

The first policy: no new map(any) or any-typed inputs on root modules. The team's terraform/ directory has a pre-commit hook (8 lines of grep) that fails the commit if any new variable block contains type = any or type = map(any). Existing instances are grandfathered, with a TODO list tracked against each module. Three of the original 14 have been converted to typed objects so far. The hook has fired four times in the six weeks since.

The second policy: every PR runs terraform plan against every environment, not just the one the contributor cares about. A matrix job in CI runs plan -var-file=envs/<env>.tfvars across all four environments and fails the PR if any of them errors. This would not have caught the original collision (plan succeeded everywhere), but it catches a different class of failure where one environment's tfvars hits an unwritten code path.

# Before: latent any-typed input
variable "feature_overrides" {
  type        = map(any)
  default     = {}
  description = "Per-environment feature flag overrides"
}

# In modules/services/database/locals.tf
locals {
  merged_flags = merge(
    local.legacy_db_flags,
    var.feature_overrides,
  )
}

# Above passes plan even when the two maps have a key
# whose value types disagree. The mismatch surfaces only
# at apply, when the receiving attribute is evaluated.

# After: typed, explicit, errors at plan time
variable "feature_overrides" {
  type = map(object({
    enabled     = bool
    rollout_pct = optional(number, 0)
    routing     = optional(string, "default")
  }))
  default     = {}
  description = "Per-environment feature flag overrides"

  validation {
    condition = alltrue([
      for k, v in var.feature_overrides :
      v.rollout_pct >= 0 && v.rollout_pct <= 100
    ])
    error_message = "rollout_pct must be between 0 and 100."
  }
}

The same variable, before and after. The lower form fails plan, not apply, when a contributor passes the wrong shape.

The structural fix took longer. A 28-input root module is not a configuration problem, it is a service-boundary problem. The team running the database stack should own a database/ root module with four inputs, not a 14-input subtree of a shared 28-input root. We split the original root into three roots along ownership boundaries (network, services, observability) using a thin terragrunt overlay for the cross-cutting variables. The split took six weeks of careful state-mv work to land without downtime. We have written more on the structural fix in the Terraform and IaC debt playbook, which covers when a shared root module starts costing more than the consistency it buys.

What we tell every team now: strong types in Terraform are not bureaucracy, they are the documentation. The half-day cost to write object({ name = string, enabled = bool, ... }) instead of map(any) buys you a plan-time failure instead of an apply-time failure, and apply-time failures land at 4:42pm on Fridays. We have stopped accepting map(any) inputs in any client engagement that involves an IaC audit, and we have not had a single contributor push back once they saw the cost.

If you are looking at a 28-input root with map(any) sprinkled through it

When your own root module is past 20 inputs

If you are reading this and your terraform/ directory has a root module past 20 inputs with several map(any) types in the input list, the failure you are heading toward is not a surprise. It is a scheduled event. The trigger will be a new contributor who does not know the implicit contract, plus one bad-enough Friday. The hardest part of cleaning it up is not the typing work itself; it is the audit of downstream consumers that have been silently depending on the loose contract for years. Two layers of modules-of-modules can hide a reference that breaks the moment you tighten the type, and your CI will not warn you because plan will keep passing right up to the apply that surfaces it.

We run these recovery and audit engagements every week. The map(any) collision pattern is the third-most-common shape we see in seed-to-Series-B SaaS Terraform repos, right after stale state lock holders and provider-version-drift cascades. It is one variant of the broader terraform apply fear problem we engage on most weeks. On a typical engagement we map every any-typed input in your root modules within the first day, prioritize them by blast radius, and either convert them in-place or split the root if the input count is the real problem. If you are looking at a Terraform root with map(any) sprinkled through it and a release window that does not forgive a 4pm apply failure, book an infrastructure review with our team and we will start with a 30-minute diagnostic call this week.

Originally published at https://infraforge.agency/insights/terraform-apply-fails-map-any-trap/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• Grafana dashboards show 'No Data' on every panel after a cluster migration, and kubectl edit fixes revert within 1-3 minutes
• Prometheus targets page is empty or stuck on a namespace that does not exist anymore
• ClusterRoleBindings you just recreated reference a ClusterRole name nobody on the team typed
• ps aux shows kworker-looking processes with elevated CPU that hold open file descriptors to a kubeconfig
• kubectl get cronjobs -A shows entries in namespaces nobody on the platform team remembers creating

Why we stopped fixing config and started looking for what was undoing it

The fix that lasted 90 seconds

The team that called us had been at this for nine hours. After a cluster migration, every Grafana dashboard was blank. The on-call had walked through the obvious things. The Prometheus datasource in Grafana pointed at port 9999. The Loki datasource pointed at port 3199. The Prometheus scrape config had annotation keys nobody recognized (prometheus_io_metrics_enabled instead of prometheus.io/scrape) and targeted a namespace that did not exist. The Grafana deployment had a config-validator init container running sleep 3600. Each one of those was a real bug. Each one of those, fixed in isolation, would revert before the next pod rolled out.

The shape of what they were describing was not a botched migration. A botched migration leaves bad state. This was bad state being re-applied. When manual kubectl edits revert in minutes, the question is no longer 'what is wrong with the manifest', it is 'what process has write access and is reconciling against a corrupt source of truth'. We told them to stop fixing config until we had inventoried every actor that could write to the cluster.

This sounds obvious written down. In the middle of an incident, with executives asking for an ETA on dashboards, the instinct is to keep patching. We have run this play enough times now to know the patching never converges. You burn three more hours and your changes still revert. The only path out is persistence-first triage.

Seven places state was being rewritten from

A kworker thread holding a kubeconfig

We started on the nodes. ps auxf on each worker showed a process named [kworker/u8:2-events_unbound]. Square brackets usually mean a kernel thread, and you learn early not to touch kernel threads. We almost moved on. The thing that snagged our attention was CPU: a real kernel worker thread on an idle-ish node should not be sitting at 12 percent. We pulled its open file descriptors.

$ ls -l /proc/$(pgrep -f 'kworker/u8:2')/fd/ 2>/dev/null | head
lr-x------ 1 root root 64 ... 3 -> /root/.kube/config
lrwx------ 1 root root 64 ... 7 -> socket:[884213]
lr-x------ 1 root root 64 ... 9 -> /opt/.reconciler/state.json
$ cat /proc/$(pgrep -f 'kworker/u8:2')/comm
kworker/u8:2-events_unbound
$ readlink /proc/$(pgrep -f 'kworker/u8:2')/exe
/opt/.reconciler/agent

Kernel threads do not hold kubeconfigs or have an exe link. This was a userspace binary with a spoofed comm name.

That was reconciler one. The same trick was on every node, with comm names rotating through plausible kworker patterns (flush-dm-0, mm_percpu_wq). We collected the binary, killed every instance, removed the systemd unit that was respawning it, and moved on. Then we did the boring sweep nobody wants to do in the middle of an incident.

• kubectl get cronjobs -A surfaced config-audit in kube-system and prometheus-metrics-federation in cattle-monitoring-system. Neither was ours. Both ran every 60 seconds and wrote ConfigMaps. Reconcilers two and three.
• systemctl list-timers on each node showed k8s-health-monitor.timer firing every two minutes against the API server with a node-local kubeconfig. Four.
• ls /etc/cron.d/ had a host cron entry running a script under /opt/.reconciler/ once a minute as a belt-and-braces backup to the systemd timer. Five.
• kubectl get mutatingwebhookconfigurations turned up the one that hurt us most, rbac-policy-enforcer, which rewrote roleRef on any ClusterRoleBinding we created. Six.
• The second mutating webhook, pod-policy-webhook, injected the config-validator init container running sleep 3600 into every pod admitted to the monitoring namespace. Seven.

Seven writers. Any one of them, left running, would have re-corrupted the stack within minutes of our fixes. Four more findings wrote nothing at all; they existed to make the cleanup slow.

• namespace-policy-webhook, a validating webhook, rejected our edits to the cattle-monitoring-system namespace object outright.
• chattr was set +i on /etc/cron.d/k8s-health and on the corrupted ConfigMap manifests staged on disk. Edits failed silently with 'operation not permitted'.
• Finalizers on the CronJobs prevented kubectl delete from completing until we patched them off.
• PodSecurity labels on cattle-monitoring-system were set to enforce a baseline that blocked our debug pods from running.

Some teams have a reconciler. This cluster had a mesh of them, each one a backup for the others, with a layer of obstacles bolted on top to slow anyone trying to unpick it. That is not a thing healthy infrastructure does; it is a thing a previous incident or a hostile takeover does. Either way, the response is the same.

The order we neutralized things, and why order matters

Why we deleted the webhooks before touching RBAC

There is a trap in this kind of cleanup. If you fix the visible problem before you neutralize the actor reverting it, you have wasted a fix and burned credibility with the room. The worst version of this in our case was the RBAC webhook. The Prometheus ClusterRoleBinding had been deleted entirely, and the deployment had been swapped to the default service account. The obvious move was to recreate the CRB and patch the deployment back to a proper SA.

We tried it once, in a scratch namespace, just to see. The CRB came back with roleRef pointing at a ClusterRole that did not exist. The mutating webhook was matching anything with 'prometheus' or 'monitoring' in the name and silently rewriting the roleRef. If we had run that against the real CRB in production with the team watching, we would have looked like we did not know what we were doing, and the fix would not have worked.

Neutralize first, then fix. RBAC and any 'monitoring'-named resource go last because the webhook would mutate them on creation.

So the order was: strip finalizers from the CronJobs, chattr -i on the immutable files, delete the three webhook configurations, suspend and delete the CronJobs in kube-system and cattle-monitoring-system, mask the systemd timer, remove the host cron entry, kill the userspace reconciler processes on every node and remove their systemd unit. Then we sat for 60 seconds and watched. No ConfigMap mutations. No Deployment patches. Quiet cluster. That was the first time in nine hours the cluster had been quiet, and you could feel the room exhale.

Restoring the observability stack once writes were ours alone

The order we put it back together

With the reconcilers gone, the config fixes were the easy part. We did them top-down by data flow: scrape config, then service routing, then the consumers.

# 1. Prometheus ConfigMap: restore annotation keys, fix namespace, drop interval
kubectl -n monitoring get cm prometheus-config -o yaml > /tmp/prom-cm.yaml
# edit: prometheus_io_metrics_* -> prometheus.io/scrape, /metrics, port
#       namespaces: [bleater-nonexistent] -> the real app namespace
#       scrape_interval: 300s -> 30s
kubectl apply -f /tmp/prom-cm.yaml

# 2. Prometheus Service: targetPort 9099 -> 9090
kubectl -n monitoring patch svc prometheus --type=json \
  -p='[{"op":"replace","path":"/spec/ports/0/targetPort","value":9090}]'

# 3. Service account and RBAC (webhooks already deleted)
kubectl -n monitoring create sa prometheus
kubectl create clusterrolebinding prometheus \
  --clusterrole=prometheus --serviceaccount=monitoring:prometheus
kubectl -n monitoring set serviceaccount deploy/prometheus prometheus

# 4. Prometheus readiness probe: port 9099 /-/healthz -> 9090 /-/ready
# 5. Loki: drop -server.http-listen-port=3199 arg, fix svc selector loki-server -> loki
# 6. Grafana: remove init container, fix probe ports, drop GF_SERVER_HTTP_PORT,
#    fix volume refs (-v2 -> base name), reset admin secret
# 7. Delete NetworkPolicy grafana-egress-restrict
kubectl -n monitoring delete networkpolicy grafana-egress-restrict

We applied these as separate kubectl operations on purpose, not a single helm rollout, so we could verify each one stuck before moving on.

After every step we waited 30 seconds and re-read the resource. Nothing reverted. We rolled the Grafana deployment, watched it come up clean with no init container blocking startup, hit the Prometheus targets page and saw 11 active up series including the application pods, then loaded a dashboard. Data. The two-minute stability window passed with no drift. We held the bridge for another 20 minutes anyway, because the team needed to see it not break more than they needed us to leave.

Persistence-first triage is now the default for post-migration observability failures

What we changed in our own playbook

We have changed how we open any incident where fixes do not stick. The first 15 minutes are no longer spent on config. They are spent on a sabotage sweep: cronjobs in every namespace (not just the obvious ones, cattle-monitoring-system bit us and we have seen it bite others), systemd timers on every node, /etc/cron.d, validating and mutating webhooks, finalizers on resources we expect to delete, immutable file attributes on staged manifests, and a ps auxf on every node with an eye on anything in square brackets that has an exe link.

We also changed how we think about kubectl edit during a live incident. If a change has to land and the cluster has any chance of having a reconciler we have not yet found, we apply through git and watch the apply, not edit the live object. It is slower by 90 seconds and saves you from spending an hour wondering why your fix evaporated. We have written more on the same instinct in our notes on Kubernetes release failures and on ArgoCD self-heal traps, which is the friendly version of this same pattern.

The non-obvious lesson from this incident is that hostile or accidental reconcilers do not announce themselves. The kworker spoof was the cleverest piece; it would have survived a casual ps. The cattle-monitoring-system namespace looked legitimate to anyone who had ever run Rancher. The webhook had a name (rbac-policy-enforcer) that sounded like something a security team would install. In each case the move that surfaced it was boring: enumerate the category exhaustively, then ask which entries the team can account for. Anything they cannot account for is the answer.

When fixes revert, the problem is not the fix

If your post-migration monitoring keeps un-fixing itself

The hard part of incidents like this is not the Prometheus annotation key or the Grafana port. Those take 20 minutes once the cluster stops fighting you. The hard part is having the discipline to stop patching and inventory every actor that can write to your cluster, especially when leadership is asking for an ETA and your instinct is to keep typing. The hard part is also knowing what the categories of reconciler are. If you have never had to look for a mutating webhook that rewrites RBAC, or a host process pretending to be a kworker, the search takes hours. If you have seen it before, it takes 15 minutes.

We run these recovery engagements every week. We have seen the kworker spoof twice this year, the cattle-monitoring-system CronJob trick three times, and the RBAC-mutating webhook in two unrelated post-migration incidents. The playbook is portable; the patience to run it before patching is the part teams in the middle of an outage struggle with, and that is usually why they call us.

If your dashboards are blank after a migration and your fixes are not sticking, book an infrastructure review with our team and we will be on a bridge with you the same day. Bring node SSH access, kubectl with cluster-admin, and a list of every namespace you can name. We will handle the rest.

Originally published at https://infraforge.agency/insights/grafana-no-data-after-migration-reconcilers-reverting-fixes/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Problem signals:

• ArgoCD shows Synced and Healthy on the Keycloak application, but downstream services return 401 on tokens they accepted an hour ago
• JWTs decoded at jwt.io are missing claims that production code depends on (groups, email_verified, audience)
• Engineers have been making emergency fixes directly in the Keycloak admin console during recent incidents and not committing them back
• The realm import ConfigMap in git has not been touched in weeks, yet the live realm has clearly changed
• Helm values for the Keycloak chart set realm import strategy to OVERWRITE or leave it unset (which defaults to OVERWRITE on most charts)

The sync that looked clean and quietly stripped six clients

ArgoCD said Healthy. Auth said 401.

Our first guess was wrong. The team had been staring at auth-service for 25 minutes when we joined the bridge, because the tokens it was issuing were obviously malformed. The groups claim was gone. The email_verified claim was gone on a different client. Surely auth-service had shipped a bad release. Except auth-service had not shipped in nine days, and the failure had started 40 minutes ago, not nine days ago.

The shape of the failure is what gave it away. Three OIDC clients had each lost a different mapper at the same moment. Auth-service had lost a groups protocol mapper. The profile service had lost an email_verified client scope mapping. The api gateway had lost role mappings for a downstream audience. Three services do not lose three unrelated pieces of OIDC config simultaneously unless something upstream rewrote all of them at once. The only thing that had touched Keycloak in that window was an ArgoCD auto-sync of the Keycloak Helm release.

We pulled the ArgoCD sync history and found the sync 41 minutes earlier. It was a chart version bump, nothing that should have changed realm content. But the chart ships a realm import ConfigMap, and the realm JSON inside that ConfigMap had not been updated in weeks. Meanwhile the live realm in the Keycloak PostgreSQL database had been edited through the admin console at least a dozen times during recent incidents. None of those console changes had been committed back to git.

So the chart redeployed the ConfigMap. The Keycloak init container read it. And the realm import ran with the strategy set to OVERWRITE. Every console change made during the previous two weeks of incident response got reverted to the stale git version, silently, with no error and no event surfaced to ArgoCD.

Diffing live realm state against the ConfigMap before doing anything destructive

Six clients had drifted and the next sync would make it worse

The first thing we did was not a fix. The first thing we did was freeze. Auto-sync was still enabled on the Keycloak ArgoCD application. If anyone touched a Helm value for any reason in the next hour, another sync would fire and a second OVERWRITE pass would run against whatever state we had managed to reconstruct. We paused auto-sync first and removed the self-heal annotation, then started the diagnosis.

# 1. Freeze the ArgoCD app so the next sync cannot fire mid-recovery
argocd app set keycloak --sync-policy none
argocd app set keycloak --self-heal=false

# 2. Pull live realm state from the Keycloak Admin REST API
TOKEN=$(curl -s -X POST "$KC/realms/master/protocol/openid-connect/token" \
  -d "grant_type=password" -d "client_id=admin-cli" \
  -d "username=$ADMIN_USER" -d "password=$ADMIN_PASS" | jq -r .access_token)

curl -s -H "Authorization: Bearer $TOKEN" \
  "$KC/admin/realms/primary/clients" | jq . > live-clients.json

curl -s -H "Authorization: Bearer $TOKEN" \
  "$KC/admin/realms/primary/client-scopes" | jq . > live-scopes.json

# 3. Extract the realm JSON ArgoCD just pushed
kubectl -n keycloak get cm keycloak-realm-import -o jsonpath='{.data.realm\.json}' \
  | jq . > configmap-realm.json

Snapshot live state before any reconciliation. The live API is now the source of truth, not the ConfigMap.

Diffing live-clients.json against the clients block in configmap-realm.json showed six clients with material differences. Two were missing protocol mappers entirely. Three had client scopes that had been removed. One had role mappings that were present in production but absent from the ConfigMap, because an engineer had re-added them by hand in the console during the firefight, before we joined the bridge, and had not committed them either. That last finding was the one that mattered most. The drift ran in both directions, and the ConfigMap was still armed: every future sync was another OVERWRITE pass waiting to delete config that existed only in the Keycloak database. This one had simply cascaded far enough to break downstream services.

Two write paths to the same realm. OVERWRITE makes one of them silently win.

Reconstructing realm state without invalidating active sessions

Why we did not re-import the ConfigMap

The obvious recovery path was to fix the realm JSON in git, commit it, and let ArgoCD re-sync. We did not do that, and the reason matters. A full realm re-import, even with the right content, runs through the Keycloak realm import flow on startup. Depending on the chart and the Keycloak version, that can rotate signing keys, drop active sessions, or invalidate refresh tokens. We had roughly 8,000 active user sessions at that moment. Forcing all of them to re-authenticate at 11pm during an active incident was not a recovery; it was a second outage on top of the first.

So we split the fix into two phases. Phase one was to restore live realm state using the Admin REST API directly, client by client, mapper by mapper. The REST API can add a protocol mapper or attach a client scope to a client without bouncing anything. Phase two was to update the ConfigMap in git to match the now-correct live state AND change the import strategy, so that the next ArgoCD sync would be a no-op rather than another OVERWRITE pass.

# Phase 1: restore each missing mapper live via Admin REST API
# Example: re-add the groups protocol mapper to auth-service client
CLIENT_ID=$(jq -r '.[] | select(.clientId=="auth-service") | .id' live-clients.json)

curl -s -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  "$KC/admin/realms/primary/clients/$CLIENT_ID/protocol-mappers/models" \
  -d '{
    "name": "groups",
    "protocol": "openid-connect",
    "protocolMapper": "oidc-group-membership-mapper",
    "config": {
      "claim.name": "groups",
      "full.path": "false",
      "id.token.claim": "true",
      "access.token.claim": "true",
      "userinfo.token.claim": "true"
    }
  }'

# Verify a freshly issued token now carries the claim before moving on
curl -s -X POST "$KC/realms/primary/protocol/openid-connect/token" \
  -d 'grant_type=client_credentials' \
  -d "client_id=auth-service" -d "client_secret=$SECRET" \
  | jq -r .access_token | cut -d. -f2 | base64 -d 2>/dev/null | jq .

Restore each mapper live, then verify the issued token actually carries the claim before moving to the next client.

We worked through the six clients in dependency order: auth-service first because every other service consumed its tokens, then the api gateway, then profile, then the rest. After each client we curl'd a fresh token and base64-decoded the payload to confirm the claim was present. Twenty-two minutes from the start of restoration, timeline-service was returning 200s again. No sessions dropped. No users re-authenticated. The Keycloak pods were never restarted.

What we changed so the next sync becomes a no-op

The one Helm value that should never be OVERWRITE

With live state correct, the dangerous artifact in the system was still the stale realm JSON in the ConfigMap and the OVERWRITE strategy that would re-apply it on any future sync. We exported the now-correct realm via the Admin API, ran it through a diff against what was in git, and committed the result. We also patched the Keycloak Helm values to set the realm import strategy to IGNORE_EXISTING.

# values.yaml for the Keycloak chart
extraEnv: |
  - name: KEYCLOAK_IMPORT_STRATEGY
    value: IGNORE_EXISTING
  # On Keycloak 22+ via Quarkus distribution:
  - name: KC_SPI_IMPORT_SINGLE_FILE_STRATEGY
    value: IGNORE_EXISTING

# For the operator/CR variant:
# spec:
#   realmImport:
#     strategy: IGNORE_EXISTING   # NOT OVERWRITE_EXISTING

IGNORE_EXISTING means the ConfigMap seeds a realm on first creation but never overwrites existing resources. This is the correct setting for any realm that humans also edit.

We re-enabled ArgoCD auto-sync and watched it run. The sync diffed clean: ConfigMap content matched live realm, import strategy was IGNORE_EXISTING, no resources were touched. Green for the right reason this time.

We changed two things in the way the team operates going forward. First, we wrote a small drift detector that runs nightly. It pulls the live realm via the Admin API, diffs it against the realm JSON in git, and posts to a Slack channel if they disagree. It is roughly 80 lines and it has caught two console-edits-not-committed in the six weeks since. Second, we now treat OVERWRITE as a forbidden value for any realm that is also editable in the admin console. If you want OVERWRITE semantics, you must also remove admin console write access for everyone except a break-glass account, because otherwise you are building a system where one of two writers silently destroys the other's work. We have written more about this category of GitOps failure in the ArgoCD and GitOps recovery cluster, and the same pattern shows up with Grafana dashboards, Argo Workflows templates, and anything else where humans and a controller both have write access to the same object.

When GitOps is silently rewriting your identity provider

If your realm config and your cluster disagree

The hard part of this kind of incident is not the Keycloak knowledge. It is recognizing that a green ArgoCD dashboard can coexist with a destroyed production configuration, and knowing which fixes preserve sessions versus which ones lock out every user in the building at midnight. The team we worked with had the Keycloak skills. What they did not have was a recovery sequence that prioritized live state capture over git reconciliation, and a clear rule about when to apply via the Admin API versus when to let ArgoCD do it.

We run these recovery engagements every week. The OVERWRITE-vs-IGNORE_EXISTING trap has hit two other teams this quarter, both on Keycloak, and we have seen the same shape on Grafana provisioning, Argo Workflows ClusterWorkflowTemplates, and a memorable case with Vault policies. The pattern is always: controller writes, human writes, controller wins on the next reconcile, nobody notices for hours.

If your identity provider, your dashboards, or any other system with human-editable state is sitting behind ArgoCD and you have ever wondered whether you are quietly losing changes, book an infrastructure review with our team and we will be on a bridge with you the same day. The first 30 minutes will tell you whether you have a drift problem, and from there we can scope a recovery that does not require kicking your users out.

Originally published at https://infraforge.agency/insights/keycloak-realm-overwrite-argocd-sync-drift/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.