The auto-dialer rang a borrower who had already paid that morning.
Not a one-off. The whole morning batch ran off a list that went stale before the first call even connected.
The bug was one line. The collections job queried overdue loans at the start of the run, held that list, and worked down it. Payments landing mid-run never invalidated anything, and bank attribution lags behind the money anyway. So someone who settled at opening got a warning notification, and then a call.
Nothing crashed. No alert fired. The job completed and the queue drained, and delivery rates looked normal. The system did exactly what it was written to do.
The cost was not technical. You can roll back a deploy. You cannot un-call someone and explain that you were wrong about their debt. That is trust spent on exactly the users you need to keep paying, and in collections it is a compliance surface, not just an apology.
What I do differently now: anything that reaches a human gets re-checked against authoritative state at dispatch time, not at list-build time. The eligibility check and the "sent" record live in one transaction, keyed by user plus action plus day, so a retry can't double-notify. And an outbound channel gets a suppression list and a rate cap before it gets features.
Batch jobs quietly assume the world holds still while they run. Anything with a side effect outside the database should probably assume the opposite.
I write these up in longer form here: https://polycratia.com/c/04a5e36e
If you run dunning or collections flows, the thing worth checking is where your final eligibility check sits: at list build, or at send...