A repayment came in, got split across the investors in the deal, and the ledger came out one cent short. Not every time. Maybe one deal in forty.
This was a P2P lending system: a borrower repays, and the payment goes out pro-rata to everyone who funded that loan. The math looks trivial. Each investor gets their share of the principal and interest, proportional to what they put in.
The bug was that each share got rounded on its own, independently, and then written. Four investors, four roundings, and the sum of the parts no longer equalled the amount that actually arrived. Sometimes a cent sat unassigned. Sometimes a cent got invented.
Nothing threw. No alert, no failed transaction. Investors saw plausible numbers. It only ever surfaced in reconciliation against what the bank actually settled, and by then it had been happening quietly for a while across many deals.
The money did not matter. A cent is nothing. What it costs you is this: once you find an unexplained cent, you can no longer tell anyone which balances are right. Every number in the system becomes a maybe until you prove otherwise, and proving otherwise means replaying history.
What I do differently now: money is integer minor units, never floats. Shares are computed as integers, and the remainder is allocated deterministically by a fixed rule rather than dropped by rounding. And the invariant (the parts sum exactly to the whole) is asserted inside the same transaction that writes them. If it fails, nothing is written and the distribution is flagged for a human.
The general lesson I keep relearning: a conservation rule that only holds on average is not a rule. In a ledger it has to hold on every single write, or you are just deferring the discovery.
I write up more of these production post-mortems on my blog, if that is your kind of reading: https://polycratia.com/c/080b3ee0
If you run ledgers, I am curious where you enforce your conservation invariants: in the database as a constraint, in the application transaction, or in a nightly job that tells you after the fact.