A verification callback showed up before my own transaction had committed.
Fiat-to-crypto onramp, real-time KYC moderation. The flow was simple: create a verification session, hand the user off to the provider, wait for the result.
Locally it worked, because locally I was the one firing the callback by hand, in a terminal, whenever I felt like it. It arrived last.
In production the provider answered faster than my own write path. The callback hit a handler built to update a row that did not exist yet. It 404'd. The provider recorded a delivery attempt against an error and moved on.
So the provider counted those users as approved while my system still showed them pending. People re-uploaded documents. The moderation queue showed state that was no longer true. Nothing crashed, nothing alerted: the only symptom was a session that would never resolve.
What changed after that: webhook handlers create, they do not update. Idempotent upsert keyed by the provider's own identifier, raw payload persisted before anything interprets it, and the originating request reconciling with whatever is already there instead of assuming it writes first.
And in dev I now fire the callback before the request that caused it. That is the default case in my tests, not the odd one.
"It works on my machine" usually means: on my machine I am the one thing producing events, and I produce them in an order that suits me. Production has independent producers, each with its own latency.
I wrote more about how I structure these provider handoffs here: https://polycratia.com/c/4be57371
If you consume provider callbacks, your handler probably needs to cope with arriving first.