A localized product title came back with a size the item never had.
Not a crash. Not a timeout. A clean, confident, well-formed title that read perfectly in the target language and described a product that doesn't exist.
That pipeline passed everything I threw at it before launch. I had hand-built an evaluation set (awkward titles, mixed-language source text, the attribute soup that comes out of retail parsers) and I could check every output, because I knew what the right answer was.
In production I don't. Catalog items arrive continuously from scrapers across different retail sources, and nobody is reading them one by one. The model's quality stopped being measurable the moment I stopped holding ground truth.
So I stopped grading the output and started constraining it. Every item now carries invariants that need no correct answer to verify: numbers, units, SKU codes and attribute counts have to survive the transformation. If a size shows up in the output that wasn't in the input, the item fails and goes to review. No opinion about language quality required.
Testing an agent asks whether it can be right. Production asks whether you would notice if it weren't. That second question describes a different system, and it's the one you have to build.
When there's no ground truth in production, where do you put the check?