The bugs that take production down are rarely in the code that does the work. They sit in the code that handles the work failing.
The mechanics are simple. The happy path runs constantly: it gets profiled, reviewed, and corrected by reality until it is honest. The failure path runs rarely, and almost never under the conditions it was written for. A one line mistake there can live for years without anyone noticing it exists.
Then it compounds, because it only ever runs while something else is already wrong.
A retry with no cap on a dependency that is timing out: one slow provider turns into your own load spike, and you finish the outage the provider started.
A catch-all around a payment call: a charge that actually settled gets written as failed, and now your ledger disagrees with the bank's. That is not a crash. That is reconciliation work for a human, days later, with money attached.
A health check that returns healthy because its own dependency probe timed out and defaulted to true: the orchestrator keeps routing traffic into a dead instance.
None of these are clever bugs. Each one is a few characters. What makes them expensive is the timing: you get the original incident, plus a second incident layered on top, and the second one is usually the one writing wrong state.
So I hold failure paths to the same review standard as money paths. Every retry gets a cap and an idempotency key. Every catch-all becomes a named exception. Every fallback separates "no data" from "could not fetch". Every default in a degraded-mode decision leans toward refusing rather than proceeding.
The uncomfortable part: this code is hard to exercise. You have to deliberately break dependencies to see it run at all. Which is why your recovery paths probably get their first real run in production...