Every test I wrote passed, and then the agent started describing a product feature that did not exist.
The pipeline adapts a retail catalog for another market: items come in from parsers, an LLM rewrites titles, attributes and descriptions, and the result goes back to the catalog.
In testing I fed it items I had picked by hand. Every field was populated. The job looked like translation...
In production the parser is the input, and a real parser returns partial items. A spec block that did not load. An attribute the source page rendered client-side. Nothing errored. The field just arrived empty.
My prompt asked for a complete, market-ready product record, so the model completed it. Material, compatibility, care instructions, written in fluent copy you cannot tell apart from the fields that were genuinely sourced.
That is what cost me. A crashing agent is a ticket. An agent that fills gaps produces output that passes human review, because nothing in the record marks which fields came from a supplier and which came from a model.
What I do differently now: input carries provenance per field, and the model may only emit fields that map to a source field that is present. Missing stays missing and gets routed to a human instead of written. Before anything touches the catalog, a validator compares the output field set against the input field set. That one check has caught more than any eval I had.
Testing told me the agent was good at the task. It told me nothing about what the agent does when the data underspecifies the task.
I write up more of these production failure modes here: https://polycratia.com/c/463cc554
If you run models over scraped or third-party data, I am curious whether you carry field-level provenance through the model call, or only validate the final shape.