Search for a command to run...
No comments yet. Be the first to comment.
If a Helm rollback of your service left a subset of pods in CrashLoopBackOff with empty database credentials while the rest keep serving, you are looking at a ConfigMap deletion race, not a bad rollba
The finance lead asked why AWS charged us $2,100 for NAT gateway data processing last month. Our normal was around $400. Nothing in the release calendar explained it: no new services, no traffic bump
The AWS bill for EKS Compute went from $4,200 to $9,800 in five days. Same cluster, no launch, no traffic event, no team asking for headroom. kubectl get nodes returned 60 m5.4xlarge instances where t
The log line was 'database schema version 23 is dirty, refusing to start' and the pod exited immediately after printing it. The team had already tried a Helm rollback to the previous chart version. Th
Infraforge Insights
20 posts
Recovery playbooks and operational lessons from real SaaS infrastructure engagements. We write about Terraform state recovery, Kubernetes release stabilization, GitOps drift, audit readiness, and cloud cost triage, each post is grounded in a specific failure mode and the sequence we ran to recover.
Audience: senior engineers, platform leads, and CTOs at seed-to-Series B SaaS companies.
If your worker is dropping jobs intermittently, sometimes crashing at boot with 'Error 111 connecting to redis:6379. Connection refused' and sometimes producing an empty result.json with no error at all, the cause is almost never a Redis performance problem. In most of the cases we get called into, it is a startup race between the worker and its dependency, hiding two quieter bugs behind it: an environment variable that silently falls back to localhost, and a job schema that got renamed on one side of the wire. This guide walks the three causes in the order they actually occur, gives you the one log grep that separates them, and shows the fix sequence that does not destroy the evidence you still need.
Problem signals:
redis.exceptions.ConnectionError: Error 111 connecting to redis:6379. Connection refused. and others start cleanlyresult.json is written on some runs, missing on others, sometimes present but with 0 processed jobs and no error in the logKeyError shows sporadic KeyError: 'job_id' traces that are being caught by a broad except block/app/output/ is owned by root:root with mode 0755, and nobody remembers whyThree failure modes wearing one costume
The reported symptom is almost always the same sentence: 'the worker is flaky, sometimes it processes jobs and sometimes it doesn't, and the logs don't really say why.' That sentence hides three separate bugs that compound. We rank them by the order we actually find them, not by how loud they are in the log:
REDIS_URL, the worker reads CACHE_URL, and the worker's client library silently defaults to redis://localhost:6379. It 'connects' to nothing and hangs or reads empty queues. Roughly 25%.job_id to id on the producer side and missed one .get('job_id') on the consumer. .get returns None, the worker's outer except Exception swallows the downstream KeyError, the job is skipped, no error is logged. Roughly 15%, and the hardest to see.The cause everyone blames first, and which is almost never it: 'Redis is slow' or 'the worker needs more memory'. We have not once found this to be the actual cause in this failure shape. If you are already sizing up the Redis instance, stop and read the next section first.
There is also a fourth cause that shows up as a hard PermissionError: [Errno 13] Permission denied: '/app/output/result.json' when the worker finally does try to write. It is real, but it is loud. This guide is about the quiet failures.
One grep that ranks the three
Before you restart, redeploy, or scale anything, capture the current state. Restarting the worker throws away the evidence that tells you which of the three you are dealing with. On docker compose, snapshot both services' logs to disk. On Kubernetes, capture kubectl logs --previous for the last crashed worker AND kubectl describe pod for its events. Do this first.
# capture before you touch anything
docker compose logs --no-color --timestamps worker > /tmp/worker.log
docker compose logs --no-color --timestamps backend > /tmp/backend.log
# k8s equivalent
kubectl logs -n jobs worker-7c9d8f5b6-x2k4m --previous > /tmp/worker.log
kubectl describe pod -n jobs worker-7c9d8f5b6-x2k4m > /tmp/worker-describe.txt
# then run the discriminating grep
grep -E 'Connection refused|CACHE_URL|localhost:6379|KeyError' /tmp/worker.log | head -40
Snapshot first, grep second. Anything that recreates the container (docker compose up --force-recreate worker, or a down followed by an up) discards the previous container's stdout under the default json-file driver. A plain restart keeps the log file, but it costs you the live process state that made the failure reproducible.
The grep output tells you which cause you have, in this order:
Error 111 connecting to redis:6379. Connection refused in the first 5 seconds of the worker's log and never again after that, it is the startup race. The worker gave up before Redis was ready.Connecting to redis://localhost:6379 (note: localhost, not the service name), or you see NO connection log at all and the queue depth reads always return 0, it is env drift. The worker never got the right URL and its client defaulted.KeyError (even one, even caught) in the traces, it is schema skew. The worker is silently dropping jobs whose payload does not match its expected shape.Order matters. The race hides the drift, and the drift hides the schema skew. Fix them in the order they surface.
Fix least-destructive first
Fix each cause with the smallest change that discriminates against the others. Do not batch the three fixes into one PR; you will not know which one worked, and the next incident will look identical. Ship them separately, verify each one, then move on.
| Step | What it does |
|---|---|
| 1. Startup race | Add a real readiness gate, not a sleep. On docker compose, use depends_on with condition: service_healthy and a healthcheck on the backend/Redis. On Kubernetes, use an initContainer that runs nc -z redis 6379 in a loop with a bounded timeout (60s), and set the worker's client to retry with exponential backoff for at least 30s after boot. sleep 10 in the entrypoint is the fix everyone reaches for; it papers over the problem until the day Redis takes 11 seconds. |
| 2. Env var drift | Rename one side to match the other in a single PR. Do NOT add a fallback like CACHE_URL or REDIS_URL; that is how you got here. Then add a boot-time assertion: if the resolved URL is localhost or empty, the worker exits with a clear error instead of connecting to nothing. This is the single change that pays for itself the fastest. |
| 3. Schema skew | Find every .get('job_id') and .get('id') across producer and consumer. Pick one name (we prefer id because it is what most queue libraries default to). Add a schema validation step at the consumer boundary that raises loudly on a missing key, and remove any except Exception: pass you find on the way. The broad except is what turned this into a silent bug. |
| 4. Output permissions | If PermissionError on /app/output/result.json is in your logs, the fix is a Dockerfile line: RUN mkdir -p /app/output && chown -R 1000:1000 /app/output before the USER 1000 directive. Do not chmod 777. If the directory is a mounted volume, set fsGroup: 1000 in the pod's securityContext instead. |
One thing to name explicitly: the tempting single-line fix of adding sleep 15 to the worker's entrypoint 'solves' the startup race in staging and hides all three bugs in production. We have watched this exact patch get merged, celebrated, and then paged the same team six weeks later when Redis restarted during a maintenance window and the sleep was not long enough. The cost of the right fix (a healthcheck plus retry) is roughly 40 lines of yaml and one afternoon. Pay it.
# docker-compose.yml, the readiness gate that actually works
services:
redis:
image: redis:7.2-alpine
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 2s
timeout: 1s
retries: 15
backend:
depends_on:
redis:
condition: service_healthy
worker:
depends_on:
backend:
condition: service_started
redis:
condition: service_healthy
environment:
REDIS_URL: redis://redis:6379/0 # one name, both sides
# no sleep, no CACHE_URL fallback
The gate is condition: service_healthy plus a real healthcheck on the dependency. condition: service_started alone only waits for the container to exist, not to be ready.
Confirm the fix by running the stack from cold at least ten times in a row and checking that the processed count equals the received count on every run. If nine out of ten pass, you have not fixed it; you have improved it. Race conditions do not get 90% fixed. For deeper patterns on this kind of intermittent K8s failure, we have written up Kubernetes release failure recovery separately.
The questions that come next
Is it safe to just add restart: always and let the worker crash-loop until Redis is up? It works, but it makes your logs noisier, it costs you real seconds on every boot, and it hides the underlying dependency graph from anyone reading the compose file later. Use a healthcheck. Save restart: always for actual transient failures in steady state.
Does this apply to RabbitMQ, NATS, or Kafka workers too? Yes, the shape is identical. The verbatim error string changes (Connection refused on RabbitMQ, dial tcp: connection refused on NATS, NoBrokersAvailable on Kafka), but the three causes and the fix order are the same. The env var drift bug is especially common on Kafka clients because KAFKA_BROKERS vs BOOTSTRAP_SERVERS is a coin flip in the ecosystem.
Can I skip the healthcheck if I use Kubernetes with readiness probes? No, readiness probes gate traffic to a pod, not startup order between pods. You still need an initContainer or an application-level retry loop for the worker to wait on its dependency. Readiness alone will not save you.
Why not just fix the broad except Exception: pass and call it a day? Because removing it in isolation will surface the schema skew as a hard crash on production traffic, and if you have not fixed the startup race first, you will not be able to tell which crash is which. Fix in the order the causes appear at boot: connectivity, config, contract.
How do I stop this from recurring? Three things, in order of ROI: (1) a boot-time assertion in every service that fails fast if a required env var is missing or resolves to localhost, (2) a schema check at the consumer boundary that rejects malformed messages loudly instead of silently, (3) a pre-merge integration test that starts the stack from cold and asserts processed == received on 100 test jobs. That third one is the single highest-value test we recommend for job-processing systems.
When the worker is dropping jobs right now
What makes this class of failure genuinely hard is not any single one of the three bugs. It is that they compound: the race gives you enough noise in the log that the drift looks like a symptom of the race, and by the time you fix the race, the drift has been silently corrupting queue state for hours, and the schema skew is dropping the recovery jobs you are firing to catch up. Untangling that under time pressure is where teams get stuck at 2 in the morning.
We have spent a lot of engineering hours in exactly this shape of incident, on Redis, RabbitMQ, and Kafka, across docker compose and Kubernetes. The pattern above is what we run on the call: capture logs first, grep to rank the causes, fix in dependency order, verify with cold-start runs. If you want a second set of eyes on it while it is happening, book a same-day infrastructure review and we will get on a bridge with your on-call engineer inside a few hours and work the sequence together. If the fires are already out and you want the pre-merge integration test and boot-time assertions in place before the next one, our platform reliability engagements cover exactly that.
Originally published at https://infraforge.agency/insights/worker-drops-jobs-intermittently-startup-race-env-drift/.
If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.