In payments, the retry is not the edge case. It's the normal case.
Every hop between you, the bank, and the card network can time out after the money already moved. The request succeeded; the response got lost. So the client retries, the webhook fires twice, the reconciliation job replays a batch — and if each of those can charge or credit again, you will double-process. Not might. Will.
The mistake I see most is treating idempotency as an API-gateway concern: dedupe the incoming request by a header key and call it done. That catches the naive double-click and nothing else. The dangerous duplicates come from inside your own system — a queue redelivering, a cron rerunning after a crash, two workers grabbing the same row.
So I push idempotency down to where the money actually changes: the ledger write. Every financial effect carries a deterministic key derived from the business event, not the transport. Applying the same key twice is a no-op that returns the original result. The API layer, the queue, and the retrying client can all be as sloppy as reality forces them to be, because the last line of defense is the one place that can't afford to be wrong.
The trade-off is real: those keys have to be stable across code changes, and picking what goes into them is a design decision you live with for years.
Where does duplicate processing bite you first — provider webhooks, your own message queue, or replayed batch jobs?