A webhook handler that returns 200 before the state is committed: one line of code, blast radius measured in days.
Nothing throws. The provider sees success, drops the event from its retry queue, and your system never learns the money moved.
Webhooks are the one place where someone else's retry policy is your durability guarantee. The moment you acknowledge, you own that event. If the acknowledgement sits above the commit boundary (before the transaction, before the queue write), you have quietly turned an at-least-once channel into at-most-once.
From both ends it looks fine. Provider logs say delivered. Your logs show a request served in a few milliseconds. The only trace is a balance that stops matching what actually settled, and that shows up during reconciliation, not during the incident.
The fix is ordering: persist the raw payload with the provider's event id, commit, then acknowledge, then process asynchronously. Deduplicate on that id, because the opposite mistake (doing all the work before replying) buys you gateway timeouts and redelivery instead.
So I treat the acknowledgement line as a boundary worth reviewing explicitly, the same way I review a money-moving transaction. Most small bugs that hurt in production are correct statements sitting in the wrong position relative to a commit.
I write more about attribution, ledgering and reconciliation seams in payment systems here: https://polycratia.com/c/2f4e9d40
Somewhere your system probably assumes a retry it never actually gets...