Most architecture diagrams stop at the happy path: a clean arrow from client to API, API to payment gateway, gateway back to database. That arrow is almost never where production breaks.
Production breaks in the half-second between a gateway charging a card and the HTTP connection dropping before your server writes the receipt. It breaks when a provider replays a webhook three times out of order, or when a retry worker wakes up after a deploy.
01 / treat idempotency as a primary key, not middleware
If duplicate delivery can corrupt a balance or send two customer messages, the bug is not the queue — it is the schema. Every external event needs a deterministic idempotency key enforced by a unique constraint right beside the state transition.
insert into ledger_events (idempotency_key, account_id, delta_cents)
values ($1, $2, $3)
on conflict (idempotency_key) do nothing;02 / close the loop automatically
A system that only logs discrepancies still forces a human to reconcile CSV files on Monday morning. Build a small, boring sweep that compares internal ledger state against external provider settlements every hour and surfaces mismatches the moment they diverge by a single cent.
“Nobody screenshots a reconciliation job. Everybody notices the day you do not have one.”
When operators trust that every missing webhook self-heals and every state transition is replay-safe, the product feels effortless from the outside — precisely because the machinery underneath refuses to guess.