An AI agent that works in testing and fails in production usually did not get worse at reasoning. It just met a second copy of itself.
In testing there is one agent, running one task, in one clean sequence. In production that same agent runs concurrently across users, retries itself on a timeout, and gets replayed by whatever queue sits in front of it. Now two runs are touching the same external state at the same time, and that state lives outside your database: in a payment provider, a CRM, a mailbox, a document workflow. You cannot wrap it in a transaction and roll it back.
This is not a new class of bug. It is the same failure mode I have been designing around in payments since 2018: an action that is safe once and destructive twice. The difference is that a payment client retries on a fixed rule you wrote. An agent decides to retry on its own, with slightly different wording, and your dedup key never matches.
What I do about it is boring, and it works. Every side effect an agent can trigger gets an operation record before the call, keyed on the business fact (this user, this invoice, this document) and not on the agent's generated text. The tool checks that record first. If the operation already exists, it returns the prior result instead of acting again. The agent is allowed to be non-deterministic. The effect layer is not.
The second thing: the tool layer refuses, loudly, rather than improvising. A tool that returns an empty result on failure teaches the agent that nothing was there, and the agent moves on. Absence and failure have to be different return values, or the model will confidently reason from a lie.
My take: reliability of an agent is not a property of the model. It is a property of the boundary you put between the model and anything irreversible. Test suites almost never exercise that boundary, because the boundary only breaks under concurrency and partial failure, which is exactly what staging does not have.
I have written more on designing those effect boundaries here: https://polycratia.com/c/6c4a27b7
If you run agents against real external systems, something already forced your tool layer to be idempotent: duplicate writes, duplicate charges, or probably a duplicate message that went out to a customer...