Nobody had edited the product, and it kept showing up as updated several times a day. The pipeline reported success every single time.
I was running an LLM pipeline that translated and adapted a retail catalog for another market. In testing it looked clean: feed in a product, get back a title and description, compare against the expected output, pass.
Production added one thing my fixtures never had: a second run over the same unchanged product.
The model is not a pure function. Same input, slightly different wording. Not wrong. Just different.
And everything downstream read "different" as "changed". The search index reindexed. Change feeds fired. The notification service fanned out to people who had subscribed to that product. Caches invalidated for no reason.
Nothing threw an exception, every stage was green. The output itself was not broken. What it cost me was write amplification through half the system, triggered by a pipeline that was technically working.
What I do differently now: the agent's output is an artifact with an explicit key, built from normalized source fields, prompt version, model version. If the key hasn't changed, I don't call the model at all, I serve the stored result. If the key has changed but the new text is semantically equivalent to the stored one, downstream still sees no change.
The test I was missing had nothing to do with output quality. It was this: run the pipeline twice over identical input, then assert the second run produces zero writes.
An agent you evaluate one call at a time tends to look fine. The failure lives in the second call.
I write more about structuring these pipelines for production here: https://polycratia.com/c/19065af8
So where did your agent first surprise you: in its output, or in what its output triggered downstream?