The agent passed every sample I fed it. Then it went live on the full catalog and started quietly rewriting product units.
I was running an LLM pipeline that adapts retail listings for another market: titles, descriptions, category wording. In testing I pulled a sample, read the outputs myself, and they were genuinely good. Right tone, real nuance, far better than a translation API.
In production the same prompt met listings I had never read. Somewhere in a long description the model decided "pack of 6" belonged in the title and rewrote the unit. It rounded a weight. It reflowed a dimension string into prose, because prose read better.
None of that is a translation error. But weight and dimensions feed carrier rate calculation, and the unit feeds price, so a language task silently became a pricing and shipping task. Re-running the prompt was cheap. The expensive part was re-deriving downstream state that had already been published.
What I do differently now: the model never writes to the system of record. It writes proposals into a staging table, and a deterministic validator decides what gets promoted. Numbers, units, currency, dimensions, SKU have to come out identical to the input. Only free text is allowed to differ. Anything touching a structured field is rejected outright, not sent for review.
The model doesn't have to be right about everything. It has to have a blast radius small enough that being wrong is boring.
Longer write-up here: https://polycratia.com/c/646bb70a
If you run an LLM inside a pipeline, I'd look hard at whatever sits between its output and your system of record.