Every item I spot-checked looked right. Then a category filter started returning a fraction of the products that belonged in it.
I had an LLM pipeline adapting a retail catalog for another market: thousands of items, titles and attributes. My tests ran per item. Feed one product in, read the output, judge it. Every sample passed, and re-reading them later I still couldn't point at a bad one.
What a single item can't show you is that the same source term came back differently across items. A material word rendered one way in this title and another way in the next, then a spec word, then a size descriptor. Each output was defensible on its own. Together they split the catalog.
Search and facets don't read like a reader. They match like a database. A term with three accepted output forms becomes three separate filter values, and a customer browsing one of them doesn't see the other two. Nothing threw an error. Nothing scored badly in testing. The catalog just fragmented quietly into pieces that no longer referred to each other.
What I do differently now: terminology gets extracted before any translation runs, pinned as a fixed input, and treated as data the pipeline owns rather than a choice the model re-makes on every call. Same source term, same output, by construction, not by hoping the model stays consistent at temperature.
And the test I didn't have is a corpus-level one: for every source term, count the distinct outputs across the whole batch and fail the run if any term has more than one. It takes one pass over the output, and it catches the class of failure no sample review will.
A per-item test can only tell you an item is fine. The thing that broke production was the relationship between items.
I wrote up more on where these pipelines leak once they meet real data: https://polycratia.com/c/1b9ba3ab
If you run an LLM pipeline over structured data, I'd want to know which invariant you check across calls. Probably every call still gets graded alone...