Your agent did not get worse in production. It just stopped being the only thing changing the world.
In testing it plans against a frozen snapshot: one caller, one request at a time, with state that holds still between the decision and the action. Every eval you run rewards exactly that.
Production takes the freeze away. Between the moment the agent reads state and the moment it acts, a human in an admin panel edits the record, or a provider callback lands, or a retry from your own queue shows up out of order. The reasoning is still correct. The world it referred to has expired.
So this is not a model quality problem, and more evals will not surface it, since evals replay a stable world too.
I stopped treating agent output as a command and started treating it as a proposal. The proposal carries the state it was computed on: record versions, the balance it read, the status it assumed. Whatever service applies the effect revalidates those preconditions inside the same transaction that commits it, and rejects the proposal if anything moved. A rejected proposal is cheap, you re-plan. A proposal committed against stale state is a wrong write you now have to go find and unwind.
The other half is the tool boundary. Every side-effecting call needs an idempotency key derived from the plan step and not from the retry attempt, or your retry logic quietly turns into a duplicate-payment generator.
Agents inherited the hard parts of distributed systems, and most of them, I think, are still tested like pure functions.
I unpack how I structure the proposal-and-revalidate layer here: https://polycratia.com/c/33165120
If your agent touches money or inventory, the thing to pin down is where you revalidate state: at plan time, or at commit time...