Refusing a snapshot a deposit watcher cannot fully read

2026-11-12 · 10 min read

Credit an on-chain deposit exactly once and you have to remember which payments you already reported, and that memory has to outlive the process. Once the memory becomes a serialised value, the restart path picks up a failure mode worse than a crash: a snapshot that is almost readable, read anyway, and a watcher that notifies a second time without noticing.

I keep running into this shape in custodial wallet and on-ramp work. Crediting is a one-way door: the balance moves, and the customer can spend it. A chain carries no matching obligation and may change its mind about what happened a few blocks back. Everything between those two facts sits in the watcher's bookkeeping, and that bookkeeping is all a restart carries. In chain-watch it is a value called WatcherState, and the decision worth writing about is not what it stores: it is what its loader refuses to accept.

A partial restore is a silent double-credit#

WatcherState holds five things: start_height, last_tip, the pending transfers that are not deep enough yet, the reported deposits still inside the reorg window, and settled, the keys of deposits buried below it. Each field fails differently if you load it loosely.

Drop settled and the watcher comes back believing it never reported anything. The chain source gives you a view of the chain as it looks right now, not a log of what was ever seen, so the next poll hands back every transfer still visible at or above the read floor, and the watcher reports all of them again. Nothing errors. The customer gets credited twice.

Misread last_tip or start_height as something other than the integer they were written as, and the read window moves. last_tip exists so a resumed watcher reads from the block after the one it stopped at, however far the chain moved on in the meantime. A height that comes back wrong does not break the loop: it widens or narrows what the next poll looks at, which is the same double-report or missed-deposit bug with a different trigger.

Accept a non-positive amount and the watcher reports a deposit for it. A credit of a negative amount is a debit. Also not a crash.

None of this is exotic. It is what you get when a loader does its best with whatever it was handed. A watcher that starts with no snapshot at all is honest about it: it starts cold, and whoever operates it knows the cost. A watcher that starts from a snapshot it understood most of is not honest about anything, and the guarantee it exists to provide is gone before the first poll.

So the loader is strict on purpose, and the strictness is all-or-nothing: WatcherState.from_dict either builds a state it fully read or raises.

Name the field, not the file#

Strictness has to buy a usable error first. A KeyError out of a nested dict comprehension tells you a key was missing somewhere in a snapshot. At three in the morning that is not enough to act on.

def _field(data: Mapping[str, object], name: str, label: str) -> object:
    if name not in data:
        raise InvalidState(f"{label} is missing {name!r}")
    return data[name]

Every read goes through it, and the label travels with the nesting level, so a transfer missing its amount says so as a transfer rather than as a dict. InvalidState subclasses ValueError, which matters for the callers who load a snapshot inside a startup path that already catches value errors around configuration parsing.

Type every scalar, including the bool that is an int#

JSON is not a type system. A value that went out through to_dict as an int can come back as a string, a float, a list, or True, depending on what touched the file in between. So every scalar gets a checked accessor:

def _as_int(value: object, label: str) -> int:
    if isinstance(value, bool) or not isinstance(value, int):
        raise InvalidState(f"{label} must be an int: {value!r}")
    return value


def _as_sequence(value: object, label: str) -> Sequence[object]:
    if isinstance(value, (str, bytes)) or not isinstance(value, Sequence):
        raise InvalidState(f"{label} must be a list: {value!r}")
    return value

The two explicit refusals are the interesting lines, and both are there because Python's type hierarchy is more generous than the data model here.

bool is a subclass of int. Without the first clause, a confirmations field holding True passes an isinstance(value, int) check and becomes a depth of one. A deposit reported with one confirmation when it really had twelve is the kind of record that reads as plausible in a support ticket and is wrong.

str is a Sequence. Without the first clause of _as_sequence, a string where a list belongs gets accepted as a list, then decomposed character by character, and the failure surfaces somewhere further in with a message about something unrelated. Refusing it at the door keeps the error pointing at the field that is actually wrong.

The deposit keys stored in settled are written as two-item lists, so they get a shape check on top of the element checks:

def _key_from_list(value: object) -> DepositKey:
    pair = _as_sequence(value, "deposit key")
    if len(pair) != 2:
        raise InvalidState(f"deposit key must hold two items: {value!r}")
    return (_as_str(pair[0], "tx_id"), _as_int(pair[1], "output_index"))

A DepositKey is the (tx_id, output_index) pair naming the transaction output a payment landed on. It is the identity the watcher deduplicates by, and the most load-bearing thing in the snapshot. A key that loses its second element is not a key.

Rebuild values through the door a chain source uses#

The other half of the design is that from_dict does not construct raw tuples of fields. It rebuilds Transfer, BlockRef and Deposit through the same constructors a live chain source feeds, so each one runs its own __post_init__ validators on the way back in:

def _transfer_from_dict(value: object) -> Transfer:
    data = _as_mapping(value, "transfer")
    return Transfer(
        tx_id=_as_str(_field(data, "tx_id", "transfer"), "tx_id"),
        output_index=_as_int(
            _field(data, "output_index", "transfer"), "output_index"
        ),
        address=_as_str(_field(data, "address", "transfer"), "address"),
        amount=_as_str(_field(data, "amount", "transfer"), "amount"),
        asset=_as_str(_field(data, "asset", "transfer"), "asset"),
        block=_block_from_dict(_field(data, "block", "transfer")),
    )

That is what keeps a snapshot from being a back door around the invariants. Transfer.__post_init__ runs the amount through _as_amount:

def _as_amount(value: Decimal | int | str) -> Decimal:
    if isinstance(value, float):
        raise InvalidTransfer("amount must not be a float; use Decimal, int or str")
    if isinstance(value, Decimal):
        amount = value
    else:
        try:
            amount = Decimal(value)
        except (ArithmeticError, TypeError, ValueError) as exc:
            raise InvalidTransfer(f"amount is not a number: {value!r}") from exc
    if not amount.is_finite():
        raise InvalidTransfer("amount must be finite")
    if amount <= 0:
        raise InvalidTransfer(f"amount must be positive: {amount}")
    return amount

Note that the loader reads amount with _as_str, not as a number. to_dict renders it with str, and reading it back as text means the Decimal gets reconstructed from the digits that were written, not from whatever a JSON parser decided a float should be. A deposit of 0.10000000 survives the round trip with its trailing zeros intact. Same reason the type refuses a float outright on the way in: a payment that is rounded once is rounded forever, and the restart path is a particularly undignified place to lose a satoshi.

Because those validators raise InvalidTransfer, the loader catches it and re-raises it as InvalidState saying the snapshot holds unusable data. The caller gets one error type for the whole operation. Anything else forces every startup path to catch two exceptions that mean the same thing, and someone will eventually catch only one of them.

The version is a refusal, not a hint#

to_dict writes version as STATE_VERSION, currently 2. from_dict compares it and raises InvalidState on any other value instead of attempting a best-effort read.

This is the part people argue about, so I will put the trade plainly. A tolerant loader that ignores an unknown version keeps the process starting after a rollback or a downgrade, and it buys that with exactly the bug this whole design exists to prevent: an older layout whose settled list is absent or shaped differently loads as an empty set, and the watcher replays. The strict loader refuses to start, loudly, with the version in the message. Refusing to start is an outage. Replaying is a double-credit. One of those a human reverses in a few minutes, and the other you reverse with an apology.

The discipline the version buys is also what keeps the format changeable later. Because the loader has never accepted anything it did not fully understand, a future layout arrives by bumping the number and handling exactly the two shapes that ever existed, instead of guessing which of several historical half-formats a file in the wild belongs to.

What the strictness buys at the boundary#

Once the snapshot is known to be complete, the constructor adopts it directly, with no defensive code of its own:

        if state is not None:
            self._last_tip = state.last_tip
            self._pending.update((t.key, t) for t in state.pending)
            self._reported.update((d.key, d) for d in state.reported)
            self._settled.update(state.settled)

And the property that produces a snapshot writes pending, revertible and settled sorted, so equal bookkeeping renders as one dict. The round trip turns into something a test can state as an equality instead of as a set of spot checks:

    snapshot = json.loads(json.dumps(watcher.state.to_dict()))
    resumed = DepositWatcher(
        chain,
        ["addr-1"],
        policy=2,
        reorg_depth=3,
        state=WatcherState.from_dict(snapshot),
    )
    assert resumed.state == watcher.state

The json.dumps and json.loads in the middle are not decoration. Testing from_dict(to_dict(state)) on live Python objects proves close to nothing, because the objects never degrade into the string-keyed, loosely typed data the loader is actually written to defend against. The serialisation step is what makes the test exercise the real boundary.

What I would do differently#

Two things in this area are not settled.

The first is the ambiguity of a defaulted argument. start_height and state are mutually exclusive, but zero is the default, so a start_height=0 passed alongside a state cannot be told apart from one that was never passed at all. It is not refused, and the snapshot's own start_height wins (the alternative would drop the floor of a resumed watcher to zero and widen every read below the height the snapshot was taken from). I think that is the right behaviour, and it rests on an argument about defaults rather than on anything the type system enforces. A sentinel would make the exclusivity real instead of documented...

The second is that settled only grows, which is what the exactly-once guarantee costs. A watcher running long enough on a busy address set accumulates keys forever, and a growing snapshot eventually becomes slow to write and large to read. Pruning it means deciding when a key is old enough that the chain can no longer surface the transfer, which is a claim about the chain and not about the watcher. I would rather have the strict loader and an unbounded set today than a clever pruning rule I cannot defend.

Strict loading looks like pedantry in a diff. It is the only place in the system where the exactly-once property can actually be enforced, because by the time the first poll runs, the watcher has already decided what it remembers.