The provider sent the same payment callback twice, and two of my workers each took one.
Locally that could not happen. My dev server ran a single process, and the handler guarded itself in memory: a set of callback IDs already seen, plus a lock around the credit step. Retried callbacks hit the set and returned early. Every test passed, including the one I wrote for duplicate delivery specifically.
In production that same service sat behind several worker processes. Each had its own copy of the set and its own lock. Two workers looked at the same callback, both found it unseen, both went ahead. Nothing crashed. Nothing got logged either, because from inside each worker it had done exactly the right thing.
The double credit was not really what it cost me. The duplicate looked legitimate from every angle (valid signature, valid payload, successful handler), so finding it meant reconstructing the sequence by hand from provider logs and my own ledger, against a user who had already noticed.
What I do differently now is narrower than it sounds. Any guard that protects money or state lives where all the processes can see it: a unique constraint on the event ID, an insert that either wins or fails. Process memory can cache, it just cannot be the thing that decides. And my local setup runs more than one worker, because a single process quietly satisfies assumptions production will not.
This pattern is worth looking for on purpose, I think. In-memory rate limiters, warmed caches, scheduled jobs that assume one runner, anything holding state between requests: correct on one process, different on four.
I write up this kind of thing in more detail here: https://polycratia.com/c/1fcb214c
If you run multiple workers, something in your codebase still quietly assumes there is only one of it...