One datetime field stored in UTC, compared against local server time. That was the bug. The blast radius: borrowers getting collection calls about money they had already paid on time.
This was a P2P lending platform with collections automation: scheduled reminders, warnings, escalation tiers, auto-dialing once a payment window closed.
The scheduler asked one question: is now past the due boundary? The boundary came out of the database in UTC. Now came from the runtime as local time. Both values were naive, so nothing crashed. No exception, no alert. The window just closed a few hours early for every loan.
The fix was small, so the fix was not the cost. The cost was that the output of that comparison was not a number on a dashboard: it was notifications already delivered, loans already moved into a collections tier, calls already placed to people who had done nothing wrong. Unwinding escalation state is manual work, and you cannot un-call someone.
That's the part I misjudged at the time. I had been grading bug severity by how wrong the code was. Severity actually tracks what the code is wired to do: a bad comparison in a report gives you a wrong figure you can recompute, and the same comparison sitting in front of an outbound action gives you consequences in someone else's life.
So now, in anything that triggers money movement, messaging, or fees: timestamps are timezone-aware end to end, time is an injected input instead of a now() call buried in business logic, and the deciding comparison gets logged with both sides and the offset. If an automated system took an irreversible action, I want to be able to reconstruct exactly which two values made it decide.
I keep longer write-ups of production failures like this one on my blog: https://polycratia.com/c/ce89bb26
If you run automated collections, dunning, or SLA timers, the thing worth tracing is where a timestamp comparison leads straight to an action you can't take back...