polycratia

Severity has almost nothing to do with how wrong a bug is. It is how many rows it wrote before anyone noticed.

Crashes are the cheap kind: they stop, they page someone, they leave a timestamp. The expensive kind produces plausible values, a price, a weight, a fee, a settled flag. Plausible passes validation, because validation checks shape and range, not truth. It passes monitoring too, because nothing is down. So it persists, and persistence is the whole mechanism that turns a one-line mistake into an incident.

Fan-out finishes the job. A derived value gets copied into a quote, an export, a document somebody already signed, a partner's cache you do not control. Each copy is a second source of truth that no longer agrees with anything, and none of them will tell you they are stale.

Then the asymmetry: the deploy fixes forward from the moment it ships. The data problem runs backwards from that moment to whenever the bug started, and usually nobody knows when that was, because the output looked fine the entire time.

So in review I don't only ask "is this correct". I ask "if this turns out to be wrong, how do I unwind it". Two cheap habits make the answer boring. Keep the raw input next to the derived value, so repair is a recompute and not a forensic reconstruction. Stamp derived rows with the version that produced them, so the affected set is a query instead of a guess.

A fix without a backfill path is half a fix. The half that stays in production is the data.

I go deeper into how I design for that in my notes here: https://polycratia.com/c/5f64ceca

Pick a derived value in your system and imagine it has been wrong for the last three months. Either you can select the affected rows today, or you are going to be reading logs...

react

$ new-project --brief

or email hey@polycratia.com