In payments, the small bug with the widest blast radius has nothing to do with arithmetic: it's returning 200 to a webhook one line too early.
A 200 reads like politeness and works like a durability claim. You are telling the provider you hold the event, so it can stop holding it for you. That retry queue is, in practice, the only redundant copy of that event you will ever have, and you don't control it. You can't replay it past the retention window, and the ack deletes it.
So the ordering matters more than the handler does.
Die before the ack, and the event comes back. Die after the ack but before the write commits, and the event is gone from the world, with nothing in your logs saying so, because the handler returned successfully.
The same bug wears a second costume: enqueueing a background job inside a database transaction. The broker takes the message right away, the worker picks it up, the transaction rolls back, and the job runs against a row that never existed.
Both are one line in the wrong place. Both fail in the one direction you cannot reconstruct afterwards: missing things leave nothing to count.
So the handler has a single job, persisting the raw payload: provider id, body, received-at. Then return 200. Processing happens afterwards, out of my own storage, where it is replayable. Dispatch after the commit, not inside it.
A bug that loses money loudly gets fixed in a day. A bug that quietly narrows what you can still recover compounds for months.
I write up more payment-side failure modes like this here: https://polycratia.com/c/4a736a3f
Probably worth knowing where your webhook path acks, before or after the commit...