Your agent doesn't fail in production because the model got worse. It fails because production won't hold still while it thinks.
A test run is a frozen world: fixtures load, nothing else writes to them, and the agent is the only actor. It reads state, plans five steps, then executes all five against exactly the state it read.
Production works the other way. Between the agent's read and its third action a user cancelled the order, a webhook flipped the payment status, a job closed the accounting period, and another instance of the same agent picked up the same task. The plan was computed against a snapshot that no longer exists. Everything after that is internally consistent and factually wrong.
This isn't an LLM problem. Order books and ledgers have had the same concurrency problem for decades. Agents wrap it in natural language, so it doesn't look like a race condition until money moves or a document gets signed against stale terms.
How I handle it: I treat the agent's plan as a proposal, not a transaction. Each action carries the version of the state it assumed. At commit time the system checks that assumption and rejects the step if the world drifted, and the agent re-plans instead of pushing through. Steps stay small enough that each one commits or fails on its own, so a rejection costs one step and not the whole run.
That turns an unpredictable failure into a boring, observable one: a stale-plan rejection with a version mismatch in the log. Boring failures are the ones you can actually operate.
Most agent evaluation still measures output quality on a static input. Almost none of it measures what happens when the state moves under the agent mid-run, which is the default condition in any system with real users.
I wrote more about how this plays out in payment and marketplace backends here: https://polycratia.com/c/f9162708
If you're running agents against live data, I'd want to know where you put the freshness check: at plan time, before each step, or only at the final write.