An agent in testing gets graded on its answer. In production it gets graded on its side effects, and nothing in a typical eval suite looks at those.
Test runs start from a clean state and stop when the output looks right. Production runs land on state that some earlier attempt already created.
The mechanism is mundane. A tool call that times out has an unknown outcome: the write may have landed, the notification may have gone out, the payout may be in flight. The model sees an error string and does the reasonable thing, which is to try again. That is how one instruction becomes two transfers.
Testing hides this because mocked tools fail cleanly. Real ones fail ambiguously.
So I treat agent tool calls the way I treat payment endpoints. Every side-effecting tool gets an idempotency key derived from the intent, not from the attempt. Intent goes into a durable log before execution and the outcome after it, so an unknown result turns into a row to reconcile instead of a decision to re-reason about. The model proposes, the ledger decides what already happened.
You also get the thing compliance asks for later: which attempt actually did what, in order.
I would not file retry-safety under reliability niceties. It is the line between an agent and an incident.
Some of your agent's actions are safe to do twice. The rest you are probably hoping it never retries...