The payment had landed. The money was sitting in the account. The user's balance still said zero.
That is how I found out my normalization function was eating leading zeros.
Incoming transfers arrive tagged only by requisites: an amount, a reference string, a timestamp. To attribute them you normalize the reference first, trim whitespace, drop separators, uppercase. Somewhere in that chain I was also casting the reference to an integer before comparing it. Reference 0041 and reference 41 became the same key.
For a long time it did not matter. A collision needs two live obligations whose references differ only by padding. Then one day there were two.
The bad match itself was small. What made it expensive was that nothing failed: no exception, no unmatched-payment alert. The matcher was confident. Aggregate reconciliation against the bank came back clean, because every cent was accounted for, it was just filed under the wrong person. It surfaced only when a user complained about their balance, and by then days of ledger entries had been written on top of the bad attribution.
Two things changed permanently after that.
Normalization became a pure, versioned function with its own test suite, and I now keep the raw requisite stored next to the normalized key forever, so any past match can be re-derived and audited instead of argued about.
And the matcher is no longer allowed to be clever. If a payment matches more than one obligation, it does not score them and pick a winner: it refuses and routes to manual review. Unmatched is a cheap problem, someone looks at a queue. Mis-matched is an expensive one, because it hides inside a total that still balances.
I write up more of these payments and reconciliation lessons on my blog, if that is your kind of reading: https://polycratia.com/c/29fc851e
If you run attribution today, the thing worth checking is what your matcher does when two candidates tie: fail loudly, or quietly choose...