A logging call threw inside a payment webhook handler. By morning I was reconciling a day of payments by hand.
The handler did the real work first, and in the right order: verify the signature, persist the event, credit the virtual account. Then it fanned out a notification and wrote an audit line, and that last part read a field that happened to be null for one payment type.
The exception escaped the handler. The framework turned it into a 500. The provider did what its documentation says it does: retried, got another 500, and after a streak of failures it stopped delivering to that endpoint at all.
Nothing in my monitoring fired. A handful of 500s sat well inside normal noise. The database was healthy. The queues were empty, because nothing was arriving. Incoming payments just stopped being attributed, and balances froze at a number that had been correct the previous evening.
Two things changed after that.
The handler now has one job: verify, persist the raw event, return 200. Anything derived from that event runs as its own retried step, where a failure is mine to fix and not a reason for a provider to disconnect me.
And I monitor absence. Time since last event, per provider, per channel, with a threshold that pages. A dependency that has quietly stopped talking to you emits no errors, and every alert I owned was watching for bad things happening. None of them watched for nothing happening.
I wrote more about the attribution and reconciliation side of this on my blog: https://polycratia.com/c/f91de547
You probably have one integration that could go silent for a full night before anyone noticed...