The first time a payment integration bit me in production, it wasn't a failed charge. It was a successful one that ran twice.
Every link in a payment chain retries: the client on a dropped connection, the queue on a timeout, the provider's webhook that fires again because your 200 got lost on the way back. None of them know the operation already happened. At-least-once delivery is the default you inherit whether you designed for it or not.
So the job was never "make sure this runs." It was "make sure running it three times leaves the same result as running it once."
The way I approach it now: an idempotency key minted at the true origin of the intent, not regenerated per retry. A persisted record of that key and its outcome before any side effect touches money. Second attempt looks up the key, sees the settled result, and returns it instead of moving funds again. The dedup has to live in the same transactional boundary as the write it guards, or you've just moved the race one layer down.
The trap is scoping the key too narrowly and treating a legitimate second payment as a duplicate, or too broadly and swallowing one the user actually meant to send twice.
Where does your idempotency boundary sit — at the API edge, the queue consumer, or the ledger write?