An agent that passes evals gets graded on its final answer. Production grades every step it took to get there.
That gap is the failure mode.
In a test run a retry is free. The agent calls a tool, dislikes the result, calls it again with different arguments, and the harness asserts on the last message. Three abandoned attempts cost nothing, because nothing outside the transcript moved.
In production those attempts are writes. A refund initiated and then corrected. A KYC case pushed to manual review, then pushed again. A notification fanned out to WhatsApp and push before the agent worked out it had the wrong recipient. Final state can be perfectly correct and the system still wrong, because the trail is a product too: webhooks fired, counterparties notified, audit rows describing a decision nobody actually made.
So I don't let the agent hold the write. It produces an intent (target, amount, reason) and a plain executor applies it once, with the idempotency key and transaction boundary I'd use for any other caller. Anything the agent touches directly is read-only or compensatable.
And I assert on the call sequence, not just the output. If a scenario passes with four tool calls where production tolerates one, that test fails.
Evals ask whether the agent was right. Production asks what it touched on the way.
I went deeper on where this breaks in payment and compliance flows here: https://polycratia.com/c/8f81718c
I'd want to know where you draw the line between what your agent may decide and what it may execute...