An agent that passes your evals and then fails in production didn't get dumber. You lost the oracle.
In testing every case ships with a known-good answer. The harness compares output to expectation and returns pass or fail. That comparison is doing most of the work, not the model.
Production has no expected answer. What it has is the state the agent left behind: a record updated, a message sent, a payment marked as handled. Nothing in the loop says whether that was right, so your first real signal is a human noticing something downstream is wrong.
So I stop verifying outputs and start verifying effects.
Every write tool gets a post-condition the agent cannot talk its way past, evaluated after the call, against the system, not against the model's summary of what it did.
Every write carries an idempotency key, because the agent's retry after a timeout looks identical to a new intent.
And the set of agent-made writes gets reconciled on a schedule, the way I would reconcile a payment flow: match what the agent claims it did against what the system actually holds, then route the unmatched to a human queue instead of letting the planner guess.
That queue is the real eval. Evals measure whether the model can be right. Reconciliation measures whether your system can tell when it wasn't, and it's the only one of the two that runs while customers are awake.
I wrote the pattern up in more detail (post-conditions, idempotent tools, reconciliation over agent writes) on my blog: https://polycratia.com/c/7aa59747
If you're running agents against live data, something catches a wrong write first: a post-condition, a reconciliation pass, or a customer. Probably worth knowing which one before it picks itself...