The agent passed every run I threw at it. Then it hit production and I couldn't answer a basic question: which output did I actually publish?
The work was an LLM pipeline adapting product catalogs for other markets. In testing I pushed records through one at a time, read the output, moved on. In production that same step ran as a queued fan-out over thousands of records.
Queues retry. That's their job. A worker stalls mid-record, the broker redelivers, the step runs again.
What my test runs hid from me: a model call is not a pure function. Same record, same prompt, different text. A retry stops being a retry: it becomes a second opinion. So some records got written twice with two different versions, last write won, and nothing in the logs admitted a second call had happened.
Nothing crashed. Nothing alerted. I found it by diffing published catalog text against what I expected to be sitting there.
The cost was mostly trust. Before I could ship anything else through that pipeline, I had to work out which records had been generated more than once.
What I do differently now: I treat a model call as a side effect with an identity, not as a computation. Before the call I derive a key from the source record version, the prompt version and the model id. If that key already has a stored result, the retry hands back the stored text and never reaches the model. The key sits beside the output, so any row can be replayed and diffed months later.
A prompt that survives manual review is not a pipeline. A better prompt rarely closes that gap: what's missing is an identity for every generation you can look up.
I wrote up more on how I structure these pipelines (input contracts, keyed outputs, replay) here: https://polycratia.com/c/2d2b4dac
If you're running agents behind a queue, the identity of a model call either lives somewhere you can point at, or a redelivery quietly generates a fresh answer...