Your agent didn't get dumber in production. It just started getting a second attempt at actions you only ever tested once.
In testing, an agent run is a clean line: one prompt, one tool call, one response, one outcome you check by hand. In production that same run is a distributed system. The model times out mid-call and the client retries. A step fails halfway and the orchestrator replays the plan from the last checkpoint. A user hits refresh and starts a duplicate. The reasoning is identical. The side effects are not.
I spent years handling this same failure class in payments, and the lesson carries over directly: the dangerous part of an operation is the write, not the decision. A charge, a payout, a status transition, a sent notification. Retry a read and you waste a little money. Retry a write and you have corrupted state that no amount of better prompting will repair.
So when I build agent pipelines, I stop treating tool calls as function calls and start treating them as money movements.
Every action that mutates anything carries a caller-supplied idempotency key derived from the intent, not from the attempt. A second call with the same key returns the first result instead of doing the work again. Actions get logged before execution, not after, so a crash leaves evidence rather than ambiguity. And anything the agent cannot safely redo (an outbound message, an irreversible transfer) sits behind an explicit confirm step owned by the system, not by the model's judgement.
The uncomfortable consequence: your agent's capability is capped by how many of its tools are replay-safe. Not by the model. I have watched perfectly good pipelines get quietly downgraded to read-only recommendations because nobody wanted to own the write path.
The other half of the gap is that tests give an agent well-formed inputs. Production gives it a truncated OCR result, a catalog field in the wrong language, a provider returning 200 with an empty body. The agent does not crash on these. It reasons over them confidently and produces a plausible wrong action, which is strictly worse than an exception, because an exception gets caught and a plausible wrong action gets executed.
My working rule: an agent is production-ready when every tool it can call is safe to call twice, and every input it can receive is one it will refuse rather than guess at.
I write up more of these production trade-offs here: https://polycratia.com/c/d7220ccc
If you run agents with real write access, I'd want to hear where your first duplicate action came from: a client retry, an orchestrator replay, or a user double-submit...