chandima
← writing / 01

why reconciliation is the product

2026-10-14 · systems · 4 min

Happy paths are easy to demo. Real payment and messaging systems are judged by what happens when a webhook arrives twice or a ledger drifts at 2 a.m.

Most architecture diagrams stop at the happy path: a clean arrow from client to API, API to payment gateway, gateway back to database. That arrow is almost never where production breaks.

Production breaks in the half-second between a gateway charging a card and the HTTP connection dropping before your server writes the receipt. It breaks when a provider replays a webhook three times out of order, or when a retry worker wakes up after a deploy.

01 / treat idempotency as a primary key, not middleware

If duplicate delivery can corrupt a balance or send two customer messages, the bug is not the queue — it is the schema. Every external event needs a deterministic idempotency key enforced by a unique constraint right beside the state transition.

insert into ledger_events (idempotency_key, account_id, delta_cents)
values ($1, $2, $3)
on conflict (idempotency_key) do nothing;

02 / close the loop automatically

A system that only logs discrepancies still forces a human to reconcile CSV files on Monday morning. Build a small, boring sweep that compares internal ledger state against external provider settlements every hour and surfaces mismatches the moment they diverge by a single cent.

“Nobody screenshots a reconciliation job. Everybody notices the day you do not have one.”

When operators trust that every missing webhook self-heals and every state transition is replay-safe, the product feels effortless from the outside — precisely because the machinery underneath refuses to guess.