Savings Plan coverage across the fleet went from 100% in June to about 41% two weeks into July, and not one thing on the platform team's dashboards moved. Node count flat. Pod count flat. p99 latency
By 08:47 the terraform plan output was up on the shared screen and nobody wanted to be the one to type apply. The summary line read 'Plan: 5 to add, 0 to change, 5 to destroy.' Every one of the 5 dest
The AWS bill for EKS Compute went from $4,200 to $9,800 in six days. Same cluster, no launch, no traffic event, no team asking for headroom. kubectl get nodes returned 60 m5.4xlarge instances where th
The first thing we saw in Loki was a fanout service log line that contained the string 'a2V5Y2xvYWstY2xpZW50' repeated about 40 times in a single minute. Base64 decode: 'keycloak-client'. The fanout s
The call did not come from our on-call rotation. It came from a customer who noticed two unrelated degradations on their side and asked why we had not paged. We had not paged because Grafana OnCall ha
The Slack ping came in at 2:14 am. Two replicas of the fanout service were stuck in Init:0/3 and the deploy queue behind them had grown to seven changes. The on-call engineer had already tried the obv
