A locale shipped green: no failed jobs, no errors. Validation passed on every record.
Weeks later someone reading the storefront asked why part of the product copy felt thinner than the rest.
The step was an LLM pass adapting catalog content for an international audience. In testing it had one path: full context in, adapted text out. In production it had two. The provider rate-limited a large batch, my retry wrapper did its job, and after a few attempts the step fell back to a reduced prompt (less context, shorter input) because a degraded result looked better to me than a dead job.
That fallback produced text that was valid. Right language, right fields, length inside the band. It passed every check I had, because every check asked "is this output well-formed" and none of them asked "which path produced it".
So degraded records merged into the catalog, indistinguishable from the good ones. The cost was not an outage: it was not knowing which rows to redo, so I had to re-run a whole slice to be sure.
Now every record carries its own provenance: primary path or fallback, attempt count, model and prompt version. A fallback gets written down as a partial failure that still owes a re-run, not as a success with a quiet asterisk.
Tests exercise the path you designed. Production exercises the paths you added to survive it...
I write more of these production notes on my blog: https://polycratia.com/c/32810fa7
If your agent degrades under load, you probably want to be able to tell, per record, which ones came out of the degraded path.