In testing an AI agent gets judged on its output. In production it gets retried, and that one difference is probably where most of them break.
A test run makes one pass over curated inputs and scores the answer. Production wraps that same pass in a queue, a timeout, and an at-least-once delivery guarantee. So the interesting case is the step that times out after the tool call already landed but before the result got recorded. The retry re-runs it, and since the model is non-deterministic, attempt two is not the same request. Slightly different arguments, different ordering, sometimes a different tool altogether.
That quietly disables the standard defence. An idempotency key derived from the request payload is useless when the payload gets regenerated on every attempt. You end up with two translated catalog entries, two notifications, two moderation decisions on one record: none of them raising an error, all of them committed.
The fix is to stop letting the model own the key. The key is the business intent: the record id and the step name the agent was invoked for, fixed before the first token is generated. The agent plans. An executor owns the key, deduplicates the side effect, and writes the result in the same transaction that marks the step complete. Anything touching money or compliance state goes through that executor and nowhere else.
Treat the agent as a planner sitting on top of an idempotent action layer, and non-determinism turns into a quality problem instead of a correctness problem. Those need very different fixes, and only one of them corrupts data while you sleep.
If you run agents in production: what is your dedup key for a tool call, the payload, or something decided before the model ran?