Savings Plan coverage across the fleet went from 100% in June to about 41% two weeks into July, and not one thing on the platform team's dashboards moved. Node count flat. Pod count flat. p99 latency
Karpenter launched 2,800 nodes over one weekend and consolidated only 180 of them, and our EC2 on-demand bill jumped 2.2x. The cause was one line added to a shared internal Helm chart on Friday aftern
The AWS bill for EKS Compute went from $4,200 to $9,800 in six days. Same cluster, no launch, no traffic event, no team asking for headroom. kubectl get nodes returned 60 m5.4xlarge instances where th
The first thing the on-call team tried was patching the status ConfigMap. Five apps showed Progressing in the platform's bleater-status object, the dashboard had been red for four hours, and somebody
The Slack ping came in at 2:14 am. Two replicas of the fanout service were stuck in Init:0/3 and the deploy queue behind them had grown to seven changes. The on-call engineer had already tried the obv
