The service I'd rewritten was correct. The rows in its table were not.
I had pulled one module out of an inherited monolith and rebuilt it as a standalone service. Clean state machine with explicit transitions, plus a test for every illegal jump. It ran about two weeks before records started showing up in states my own code could not produce.
The rewrite itself held up. What I had missed: the monolith still had an admin panel and a nightly job writing to those same tables directly, and both predated the module I replaced. My state machine was true only for traffic that came in through my service. Everything else wrote whatever it liked, and then my retry logic woke up and acted on that state as if it had produced it.
Nothing went down. What it cost me was a couple of weeks of not trusting my own invariants, which is probably the worse outcome: you start adding defensive branches instead of fixing the boundary.
So now, before any rewrite, I answer five things in writing first.
Who else writes to this data, outside the code I'm replacing?
What states already exist in production that the new code has to accept, not just produce?
Which callers depend on the current shape of the output?
What happens during the window when both versions write?
What's the rollback path once both of them have?
A rewrite replaces code. It does not replace the other authors of the system, and those are the ones who will quietly disprove your design.
I'm writing up more of these migration post-mortems over at https://polycratia.com/c/7fc186bc
If you've done a monolith-to-services split, I'd like to hear what turned out to still be writing to your tables after you thought you owned them.