Production field guide

Production Readiness Checklist

Every check came from something that broke on a system serving real traffic. Use the ones that fit your architecture before a launch, migration or on-call handover.

20: Checks
4: Risk areas
4: Incidents

0 of 20 complete

Download PDF

Risk area 01

Message handling and acknowledgment

Queues, brokers and event buses where work can disappear after a component reports success.

The acknowledgment happens after the work, not after the receive.Anything that acknowledges on delivery has converted a crash into silent data loss.
The consumer confirms execution rather than assuming it.A downstream call that returned 200 is not the same as a downstream that committed.
Redelivery is bounded.Unbounded retry turns a loss bug into a hot loop, which is worse and louder.
Transient and terminal failures are handled differently.Retrying a malformed payload forever is not resilience.
Dropped work is observable.If the only symptom is a number that should have gone up and did not, you will find out from a customer.

Risk area 02

Request capacity and dependencies

Services with a bounded pool of workers, threads or connections serving requests.

Every outbound call has a deadline.The deadline is smaller than the caller's. A call with no timeout is a worker you have lent out indefinitely.
Retries have a budget, not just a count.Three retries per request across three layers is twenty-seven calls.
Worker occupancy is a graph you can look at.Latency and error rate will not tell you that the pool is full.
Readiness does not depend on every downstream.A probe that calls everything can convert a partial outage into a total one.
You know what happens at saturation.Shedding load is a decision. Queueing until the deadline is also a decision, made by default.

Risk area 03

Replacing a live data path

Migrations where an old and a new system must both be correct for a while.

The migration transfers authority, not just data.Name the moment the new system starts being the one that decides.
The shadow path compares decisions, not payloads.Two systems can disagree on every field and agree on every outcome, or the reverse.
The target is zero unexplained differences.Explained differences are progress. Unexplained ones are the whole risk.
Coverage is measured across scenarios, not request volume.A million requests through one code path proves one code path.
Rollback does not require a deploy.If reverting means shipping, you will not revert at 3am.

Risk area 04

Rollout and schema coordination

Applications running more than one replica while one-shot work or schemas change.

One-shot work runs once.A Deployment starts multiple copies of a long-running process. A migration is not that shape.
Coordination is not compatibility.A lock stops two migrations colliding. It does nothing about old pods serving traffic against a new schema.
Schema changes are expand-and-contract.Add compatible, deploy both-aware code, backfill, switch, then remove.
Waiting is logged as waiting.Expected contention that surfaces as a crash loop will be debugged as an outage.
A failed migration stops the rollout.Do not let the application start against a schema it does not understand.

How to use it

Find the checks where the honest answer is “I do not actually know.”

Work through the relevant sections with someone who disagrees with you. The value is not in completing every box. It is in finding the three or four assumptions most likely to page you.

Get the next field note →