A worker restarted mid-batch. The queue redelivered a few hundred catalog items, and the pipeline translated all of them a second time.
Same source rows, same prompt, different output text.
Nothing errored. I had run that batch end to end in staging many times, though only ever one clean pass at a time: no redelivery, no restart. The agent step passed the tests I had, because I had not asked it the same question twice.
Production asks twice. Queues redeliver, workers die mid-batch, and sometimes someone replays a failed job.
Translation quality held up fine. The real damage was that items already reviewed and published came back changed. The human moderation pass was invalidated for rows that had not been edited, and the diffs stopped telling me what had actually changed.
What I do differently now: I treat an agent call as a write, not a function call. Each one gets a key (item id, source content hash, prompt version). The output is stored under that key. A retry reads the stored result instead of generating a new one. New generation happens only when the key changes, which means a real source edit or a deliberate prompt bump.
Non-determinism on its own is fine. Non-determinism plus retries is what breaks a pipeline, and retries are not optional in production.
If you run LLM steps in a pipeline, you want to know what your key is, and whether a replay gives you back the same artifact...