The adapted catalog text read perfectly. Schema valid, every field filled, and the logs were clean.
It reached me as a support ticket, not as an alert: a product's dimensions had been quietly rewritten into units that suited the target locale, and the conversion was wrong. Fluent and confident, and incorrect.
Why it passed testing was embarrassing once I saw it. During testing I read the output. Every item, by hand, because there were few enough to read. Production took that reviewer away and nothing picked up the job. The pipeline had never been validated against the source record: it was validated against my attention.
LLM output fails in a way ordinary code does not. It stays well-formed while becoming wrong. Nothing raises, nothing retries, so every check built on shape keeps reporting success.
What I do differently now: for anything factual in the input, numbers, units, model codes, attribute values, the check is whether I can find that value, unchanged, in the output, and not whether the text reads well. A mismatch fails that item and routes it to review instead of publishing it.
And I keep a fixed share of production output in front of a human permanently, not only during rollout. A pipeline nobody reads has no error signal at all.
If you run LLM work in production, the thing I would want to know is what tells you an output was wrong, and how long it takes to tell you...