polycratia

"It works on my machine" is almost never about the machine. It is about concurrency of one.

Locally you are the only caller. One process, one worker, no retries, no provider hitting your webhook twice because the first response timed out, no second container holding a lock the first one thinks it owns. Any bug that needs two things to happen at the same moment is invisible by construction: not undetected, undetectable.

The shape is usually check-then-act. The handler reads a payment, sees it has not been credited, writes the ledger entry. Serially that is correct, and it stays correct through every local run you do. Put two concurrent callbacks on it and both reads finish before either write lands, so the balance moves twice. A process-level lock tells the same story: fine while there is one process, quietly a no-op the moment you run two replicas. Nothing in your environment can warn you, because your environment never had a second replica.

So I stopped treating this as an environment-parity problem. I push the invariant down to where concurrency cannot route around it: a unique constraint on the idempotency key instead of a lookup, conditional updates that fail when the state has already moved, state machines that reject an illegal transition rather than code that checks first and trusts the gap. Containers give you identical dependencies. They do not give you identical arrival order.

So the parity I go after is the concurrency level rather than the environment...

I write up more of these payment-side failure modes here: https://polycratia.com/c/f011b2ad

Your first real one probably surfaced in one of the familiar places: duplicate provider callbacks, parallel retries, or a lock that stopped working the day you scaled out.

react

$ new-project --brief

or email hey@polycratia.com