Recovery playbooks and operational lessons from real SaaS infrastructure engagements. We write about Terraform state recovery, Kubernetes release stabilization, GitOps drift, audit readiness, and cloud cost triage, each post is grounded in a specific failure mode and the sequence we ran to recover.
Audience: senior engineers, platform leads, and CTOs at seed-to-Series B SaaS companies.
Savings Plan coverage across the fleet went from 100% in June to about 41% two weeks into July, and not one thing on the platform team's dashboards moved. Node count flat. Pod count flat. p99 latency
Karpenter launched 2,800 nodes over one weekend and consolidated only 180 of them, and our EC2 on-demand bill jumped 2.2x. The cause was one line added to a shared internal Helm chart on Friday aftern
119 ephemeral preview namespaces should have been reaped between 2am and 6am. None were. The teardown cron had been failing on x509: certificate has expired or not yet valid for six hours before anyon
By 08:47 the terraform plan output was up on the shared screen and nobody wanted to be the one to type apply. The summary line read 'Plan: 5 to add, 0 to change, 5 to destroy.' Every one of the 5 dest
If a Helm rollback of your service left a subset of pods in CrashLoopBackOff with empty database credentials while the rest keep serving, you are looking at a ConfigMap deletion race, not a bad rollba
If your worker is dropping jobs intermittently, sometimes crashing at boot with 'Error 111 connecting to redis:6379. Connection refused' and sometimes producing an empty result.json with no error at a
The finance lead asked why AWS charged us $2,100 for NAT gateway data processing last month. Our normal was around $400. Nothing in the release calendar explained it: no new services, no traffic bump
The AWS bill for EKS Compute went from $4,200 to $9,800 in six days. Same cluster, no launch, no traffic event, no team asking for headroom. kubectl get nodes returned 60 m5.4xlarge instances where th
The log line was 'database schema version 23 is dirty, refusing to start' and the pod exited immediately after printing it. The team had already tried a Helm rollback to the previous chart version. Th
