# How one stuck PDB doubled our EKS autoscaler bill in five days The AWS bill for EKS Compute went from $4,200 to $9,800 in five days. Same cluster, no launch, no traffic event, no team asking for headroom. kubectl get nodes returned 60 m5.4xlarge instances where the working set is normally 22, and forty of them were sitting under 8% CPU. The cluster autoscaler had scaled up overnight for a batch burst that finished by 6 am; it had never scaled back down. **Problem signals:** - Your AWS bill for EKS Compute jumps without a matching traffic or deploy event - kubectl get nodes returns dozens more nodes than the working set, most at low CPU utilization - Cluster autoscaler logs repeat 'cannot delete: pdb blocking' and keep naming the same PDB - A single PodDisruptionBudget in the cluster shows ALLOWED DISRUPTIONS = 0 while others are 1 or higher - A workload somewhere in the cluster has a pod stuck in ImagePullBackOff that nobody is watching ## The $9,800 bill and 60 nodes at 8% CPU *The Friday morning spike* The bill spike was five days old by the time the AWS Budgets alert fired. It tripped at $9,000 for the billing period; the same window the previous month was $4,200. Nothing had shipped that week, CI had been quiet since Wednesday afternoon, and no product team was asking for extra capacity. Compute Optimizer had already flagged the cluster's managed node group as significantly overprovisioned. kubectl get nodes returned 60 m5.4xlarge nodes; the normal working set for this cluster is 22. Forty of the extras were sitting under 8% CPU, hosting kube-system DaemonSet pods and a single Deployment nobody had touched in months. The math is straightforward. m5.4xlarge on-demand in us-east-1 is $0.768 an hour. Forty extra nodes for six days is $4,400 of surprise burn, which matches the bill delta within a few hundred dollars. The overnight autoscale had happened on the previous Saturday to run a scheduled batch job; the batch finished by 6 am, and then the cluster autoscaler just stopped scaling down. Six days of that pattern is how a bill doubles. ## Why HPA runaway and traffic burst were both wrong *Two theories that died fast* Two theories came up on the bridge in the first five minutes. The obvious one was HPA runaway: a Horizontal Pod Autoscaler misreading its metric and scaling a Deployment to hundreds of replicas, forcing the cluster autoscaler to hold capacity to place them. The second was a traffic burst: some overnight event pushing real request volume up, driving replicas up, then subsiding but leaving the nodes. Both died fast. kubectl get hpa --all-namespaces showed every HPA sitting comfortably below its ceiling, none within striking distance of a scale trigger. Prometheus request-rate graphs for every ingress-fronted service were flat across the whole week. When we summed pod counts across the cluster, we got 342, roughly what a normal Friday looks like, and nowhere near the 900+ pods it would take to justify 60 m5.4xlarge nodes at typical density. The demand side was fine. Something on the supply side was refusing to shrink. ## The cluster-autoscaler log that pointed at exactly one PDB *One PDB, named in every log line* We went to the cluster autoscaler's own logs, which is where CA tells you plainly why it will not do the thing you want. kubectl -n kube-system logs deploy/cluster-autoscaler --tail=200 returned the same shape of line, forty times per scale-down cycle, each naming a specific node, all of them naming the same PDB. ``` I0313 09:32:04.881 static_autoscaler.go:428] Scale down status: lastScaleDownFailTime=2026-03-13 09:22:04 I0313 09:32:04.892 cluster.go:151] Fast evaluation: node ip-10-42-14-88.ec2.internal, cannot delete: pdb data-platform/nightly-rollup-pdb blocking eviction of pod data-platform/nightly-rollup-7f8b9c5d4-4kx2m (currentHealthy=39, desiredHealthy=39) I0313 09:32:04.895 cluster.go:151] Fast evaluation: node ip-10-42-15-19.ec2.internal, cannot delete: pdb data-platform/nightly-rollup-pdb blocking eviction of pod data-platform/nightly-rollup-7f8b9c5d4-9h4tn (currentHealthy=39, desiredHealthy=39) I0313 09:32:04.897 cluster.go:151] Fast evaluation: node ip-10-42-15-33.ec2.internal, cannot delete: pdb data-platform/nightly-rollup-pdb blocking eviction of pod data-platform/nightly-rollup-7f8b9c5d4-b7q9r (currentHealthy=39, desiredHealthy=39) ... (37 more lines, all identical shape, all naming data-platform/nightly-rollup-pdb) ... I0313 09:32:05.114 scale_down.go:1052] Scale down considered 40 nodes, 0 removable ``` *Every blocked node in every iteration named the same PDB. That was the whole answer in the first log grab.* We cross-checked against the cluster's full PDB inventory to confirm no other PDB was involved. ``` $ kubectl get poddisruptionbudgets --all-namespaces NAMESPACE NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE api-gateway api-gateway-pdb 2 N/A 3 47d auth auth-service-pdb 1 N/A 2 47d data-platform nightly-rollup-pdb 39 N/A 0 213d data-platform warehouse-writer-pdb 1 N/A 2 47d payments payments-api-pdb 2 N/A 1 47d search search-indexer-pdb 1 N/A 2 47d ``` *Six PodDisruptionBudgets. Exactly one at zero allowed disruptions. Every other PDB had two or three to spare, which is the healthy state.* The outlier was data-platform/nightly-rollup-pdb, and it was 213 days old, while the other five had all been re-created seven weeks earlier as part of a cluster upgrade. Something about this specific PDB had been quietly wrong for a long time. We looked at the underlying workload. The 'nightly-rollup' service started life as a CronJob and had been reimplemented at some point as a 40-replica Deployment with soft pod anti-affinity, running continuously to feed a downstream aggregation service. Its PDB carried minAvailable: 39, which is the standard 'allow one disruption at a time' setting for rolling updates. Fine when the workload is healthy. kubectl get pods -n data-platform -l app=nightly-rollup told the rest of the story. Thirty-nine pods Running, one pod stuck in ImagePullBackOff. Six days earlier, someone had pushed a rolling update with a container image reference that pointed at an ECR registry we had migrated away from during a project the previous quarter. Every other workload had been re-pointed at the new registry during that migration; this one Deployment had been missed, and nobody had noticed because it 'just worked' for months. The instant one pod dropped out of Ready, healthy went from 40 to 39 against a minAvailable floor of 39, and allowedDisruptions clamped to zero. From that moment on, no node hosting a nightly-rollup pod could be drained: evicting one would take healthy below the floor, and cluster autoscaler is careful about that. Forty pods, forty nodes, one stuck PDB. Each nightly-rollup pod requested about 400m CPU and 1.5 GiB of memory, which is a rounding error on an m5.4xlarge, but the pod's presence plus the PDB at zero meant the node was unremovable regardless of how much headroom the rest of the box had. Forty nodes at 5-8% CPU, all of them technically pinned by one broken image reference. ## Why we deleted the PDB instead of pushing the fixed image *Delete the PDB, or fix the image* Two paths out. Fix the underlying pod: patch the Deployment's image reference, let the rolling update roll forward, healthy count returns to 40, allowedDisruptions goes positive, CA is free. Or delete the PDB directly: no PDB, no eviction block, CA can drain nodes. We almost went with the image fix, because that is the 'correct' answer in an abstract sense; the PDB is doing what it was written to do. The problem was who owned the Deployment. The team that had originally built the nightly-rollup service had been reorganized eighteen months earlier, and the current owning team had inherited it in a spreadsheet handoff and had never shipped a change to it. To fix the image correctly we needed either to push the correct image to the old ECR registry (which required someone with write access on an account we were sunsetting) or to patch the Deployment to point at the new registry (which required knowing whether the image tag existed there and had feature-parity, which we did not, at 9:30 am). The PDB, on the other hand, was clearly orphaned. Its author was gone. Its minAvailable: 39 was arithmetically fine for a 40-replica Deployment during healthy operation but it meant one broken pod would freeze the entire fleet, which is exactly what had happened. Deleting the PDB did not remove any replicas or affect the currently-running pods; it just removed the eviction guardrail. ```bash kubectl delete poddisruptionbudget -n data-platform nightly-rollup-pdb # poddisruptionbudget.policy "nightly-rollup-pdb" deleted ``` *One command. The gate is the PDB; remove the gate.* Twelve minutes later, cluster autoscaler started removing nodes. It took about ninety minutes to work through the fleet. The nightly-rollup pods that got evicted rescheduled onto the remaining nodes without issue because the anti-affinity was preferredDuringSchedulingIgnoredDuringExecution (soft), which meant CA could consolidate the fleet onto fewer nodes when memory allowed. The pending pod stayed pending, because the underlying image reference was still wrong, but a single pending pod is a monitoring problem, not a bill problem. By 11:30 am the cluster was back at 24 nodes. We paged the data team's on-call to fix the image reference at their leisure the following Monday. The bill for that Friday's Compute came in at $760, back inside the normal envelope. The instinct in the first thirty minutes had been to kubectl delete node on the idle boxes. That would not have worked, and not for the reason you might expect. kubectl delete node only drops the Node object from the API server; the kubelet on that instance re-registers within seconds and the node comes back with the same name. The EC2 instance is never terminated and the orphaned PDB is untouched, so the spend does not move. Kicking nodes was never going to touch this. The PDB is the gate. Remove the gate or fix the gated pod; kicking the nodes directly is a way to spend money without changing anything. ## The Kyverno policy and PDB alert that closed the gap *The rules we shipped the next week* The postmortem produced three changes we shipped inside the week. The first was a cluster-level admission policy that rejects new PDBs unless they carry an owner annotation and an expires-at annotation. If a PDB is going to have the power to freeze half a cluster, someone needs to own it and someone needs to argue for renewing it. Kyverno was already installed for other reasons, so the rule was about thirty lines. ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: pdb-lifecycle-required spec: validationFailureAction: Enforce rules: - name: require-owner-and-expiry match: any: - resources: kinds: - PodDisruptionBudget validate: message: "PDBs must carry infraforge.io/owner and infraforge.io/expires-at annotations. See runbook: https://internal.infraforge.io/rb/pdb-lifecycle" pattern: metadata: annotations: infraforge.io/owner: "?*" infraforge.io/expires-at: "?*" ``` *Kyverno ClusterPolicy blocking any PDB that arrives without an owner or an expiry.* The exemption path for genuine long-lived PDBs (data-plane primary databases, for example) is to set infraforge.io/expires-at to a far-future date and put the owning team alias in infraforge.io/owner. That does not prevent the specific failure we hit, but it does mean that in eighteen months when the next team reorganization happens, we can grep for the PDBs whose owners no longer exist before they become orphaned. The second change was a Prometheus alert that fires when any PDB has allowedDisruptions == 0 for more than thirty minutes. Not five, not sixty. Thirty is long enough that a healthy rolling update has finished but short enough that a stuck one still gets caught inside the same business day. The third was a workload audit that the source scenario also recommends. Over the following week we ran kubectl top pods on a rolling seven-day window against every Deployment with minReplicas > 1, and dropped the floor on five workloads whose real utilization was under 30%. Two of them we took from minReplicas: 3 to minReplicas: 1. That work did not fix the incident (the PDB was the actual bug), but it lowered the cluster's headroom baseline enough to knock another two nodes off the normal working set. On the observability side we replaced the flat AWS Budgets threshold with anomaly detection at the EKS Compute line item, tuned to fire on a 30% week-over-week jump. The flat $9,000 threshold caught this incident five days late. The anomaly rule would have caught it inside twenty-four hours. ## When your EKS bill doubles without a traffic event *When this is happening to you* Bill spikes without a matching traffic event almost always trace to a controller doing exactly what it was configured to do, in a way the config never anticipated. Cluster autoscaler is the most common source of this specific flavor, because its downscale logic is deliberately conservative around PodDisruptionBudgets; a single PDB with zero allowed disruptions can pin nodes indefinitely, and if the PDB is stuck on a pod nobody currently owns, nobody notices until the bill does. HPA misconfiguration, Karpenter provisioner drift, and stuck node-termination lifecycle hooks are the other three variants we see most often in the same shape. We run recovery engagements with this exact shape most quarters. The fastest path to the answer is usually the cluster autoscaler logs, not a Prometheus dashboard, because CA will just tell you what it is refusing to do. If your EKS bill doubled inside a week and Compute Optimizer is flagging the node group as overprovisioned, we can be on a bridge with your platform team the same day. [Book an infrastructure review](https://infraforge.agency/review/) and we will start with a 30-minute diagnostic call this week; the more artifact you can share ahead of the call (bill line item, kubectl get nodes output, and a kubectl -n kube-system logs deploy/cluster-autoscaler --tail=500 grab), the faster we can name the specific blocker. If you are earlier in the shape (rising EKS bill, no obvious cause yet, no acute page), the [Kubernetes and CI/CD stabilization](https://infraforge.agency/kubernetes-cicd/) work we do covers the workload-audit and admission-policy side of what this article described. We have written more on the general pattern of cost spikes driven by controller behavior in the [cloud cost spikes](https://infraforge.agency/problems/cloud-cost-spikes/) problem write-up. --- Originally published at [https://infraforge.agency/insights/eks-autoscaler-stuck-pdb-cost-spike/](https://infraforge.agency/insights/eks-autoscaler-stuck-pdb-cost-spike/). If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — [see /review](https://infraforge.agency/review/).