The reconciliation job ran clean on my laptop for a week. Staging matched every payment. Then it ran one night in production and reported settled payments with nothing to attribute them to.
The matching logic was fine.
My laptop was on local time. The container was on UTC. The job cut the day at midnight, and midnight was a different instant in each place. Payments landing in the last hours of the local day fell into the next UTC window, so they got compared against a settlement file that hadn't seen them yet.
That unmatched set isn't a bug report. It's a question somebody has to answer by hand: did money arrive that I can't account for? It costs someone an afternoon, and it costs everyone a little trust in the numbers, which is the expensive part. Trust in reconciliation output is the whole reason the job exists.
What I do differently now.
No time arithmetic reads the host's zone anymore.
Day boundaries are an explicit property of the rail: a configured cutoff and zone per source, stored alongside the window, so a report can tell you which boundary produced it.
And the test suite runs with the timezone set to something deliberately wrong. If results change, the code is reading something it shouldn't.
Environment differences rarely announce themselves. They don't crash: they shift windows, round differently, order rows differently. The output still looks plausible, and that's why it reaches someone's eyes before it reaches your logs.
I write up more of these production-first patterns on my blog: https://polycratia.com/c/9178de1a
Host-level defaults bite quietly, and timezone is only the one that got me: locale and collation are sitting right there too...