Overnight the pipeline died halfway through a catalog batch. It restarted on its own, and I took that for the good news.
The job was an LLM pipeline adapting product catalogs for another market. Per item it ran a few steps in order: normalize the source fields, translate, adapt the category, write back. In testing every run went start to finish, and nothing died in the middle.
In production the provider timed out mid-batch and the worker came back up. The agent picked the same items back up and planned from the beginning, except the world was no longer the one it had planned against. The first two steps had already landed. So it ran them again on top of their own output: already-adapted text got adapted a second time. The write-back it never reached was still missing. Nothing raised an error. The batch reported success.
The cost was a quiet second pass over text that was already correct, caught only because a reviewer noticed the same source term coming out differently.
What I do differently now: the plan is not state. Each step writes an entry to a durable step log keyed by item and step, each step is idempotent, and resume reads the log instead of asking the model what to do next. The model still chooses steps. The log decides which ones are still owed.
A test run gets to finish. Production kills your process in the middle and asks you to prove what you had already done.
I write about this kind of thing in more depth here: https://polycratia.com/c/df6c8411
If you run agents on real data, your resume path gets its truth from one of two places: a log, or the model....