The agent passed the tests I wrote for it, so overnight I let it run across a whole product category.
By morning it had produced pages that were fluent, confident and quietly wrong.
Dimensions rewritten into other units. A model number tidied up into something cleaner. A size chart paraphrased into prose that read well and told the customer nothing precise.
Nothing failed. No exception, no retry, no alert. Each output looked good on its own, which is exactly how I had tested it: by reading it myself, one at a time.
That was the whole difference between my test runs and production. In testing, each output had a reader. In production the output goes straight into the database and onto a page, and most of it stays unread. Quality judgment doesn't scale, and a plausible sentence never throws.
So I stopped treating model output as text to review and started treating it as a write to validate. The parts that have to survive untouched (identifiers, numeric values, units, measurements) get extracted before the call and asserted after it. If a value that was supposed to be copied comes back changed, the row isn't written: it goes to a queue for a human. The model gets the prose. It doesn't get the facts.
Testing tells you the agent can do the task. Production asks a different question: what happens to the output that stays unread.
I wrote more about how I structure those invariants, and where they belong in the pipeline, here: https://polycratia.com/c/15a8b2ad
If you're running an agent against real data, it's probably worth knowing which wrong field you would actually notice, and which one would just quietly ship...