polycratia

Idempotency belongs in the ledger, not in every caller

· 7 min read

A payment callback that arrives twice must not move money twice. The usual answer is a unique constraint on the entry id, which turns the second attempt into an error, and an error is not the same thing as knowing the first attempt landed. That gap is where duplicate postings come from.

I have been building payment and custodial systems since 2018, and the shape of this bug has not changed once in that time. A provider redelivers a deposit callback. A queue redelivers what it already delivered. An HTTP client gives up at thirty seconds on a write that committed at thirty-one. In every one of those cases the caller ends up asking the ledger for the same movement a second time, and the ledger has to answer without moving anything again.

A refusal is not an answer#

The first defence you reach for is identity. Give the entry a deterministic id, put a unique index behind it, let the second write fail. ledger-core does exactly this at the journal level: a duplicate id is rejected, and Journal.extend applies the same rule to a batch, so if any entry in it is refused, none of the batch is kept.

python
from datetime import datetime, timezone
from decimal import Decimal

from ledger_core import Account, AccountType, Entry, Journal, Money

cash = Account("cash", AccountType.ASSET, "EUR")
customer = Account("customer:42", AccountType.LIABILITY, "EUR")

deposit = Entry.transfer(
    entry_id="e-1",
    occurred_at=datetime.now(timezone.utc),
    debit=cash,
    credit=customer,
    amount=Money(Decimal("25.00"), "EUR"),
    memo="card deposit",
)

journal = Journal()
journal.append(deposit)
journal.append(deposit)  # refused: the id is taken

Necessary rule, insufficient one. Identity protects the journal and does nothing for the caller. When the second append fails, the caller is holding an exception that fits at least three different worlds: my earlier attempt landed and this is my own retry, my earlier attempt never ran and something else is sitting on this id, my earlier attempt landed but wrote something other than what I am holding now. The exception does not separate them, and the caller needs that separation before it can answer the provider with a success.

So the caller writes recovery code. Catch the error, read the entry back, compare it field by field against what it meant to write, decide whether the write counts as done. That is idempotency logic, and once it exists it exists in the deposit handler, and then again in the reconciliation job that reposts stuck movements, and again in the admin tool an operator uses at two in the morning. Three copies, each subtly different, each written under different pressure. The one in the admin tool is the one that will be wrong.

The key is the question the caller asked#

Stop treating the retry as a collision and treat it as a repeated question. The entry id says which posting this is. The operation key says which write request this is, and the same request asked twice is one write.

python
from ledger_core import Operations

operations = Operations(journal)

def record_deposit(event_id: str, entry: Entry) -> Entry:
    return operations.post(f"psp-deposit:{event_id}", entry)

post writes the entry the first time and, every time after that, returns the entry the first call wrote, read back out of the journal rather than out of a cache of whatever the caller happened to pass in. post_many does the same for a batch under one key, under the journal's all-or-none rule.

The return type carries the design. post hands you back an Entry, not a boolean and not None. A caller written against it cannot branch on whether the write was new, because that fact is never offered. It gets the postings that are in the journal under its key, which is the only thing it ever needed in order to answer the provider. Retry-awareness stops being a code path and becomes the ordinary path.

Same key, different work#

A key that only stored "this key was used" would be a trap. Keys get derived badly: someone keys a payout by user id and calendar day, someone reuses a request id across two movements, and a naive dedup table would answer the second, genuinely different write with the first write's entries and no complaint. Money would go missing quietly.

Operations stores a fingerprint alongside the key: a stable digest over each entry's id, timestamp, memo, correction target, every posting's account, amount, currency and side, plus the entry's metadata in sorted order. The amount is normalized before it is hashed, so a retry that rebuilt 25.0 where the first attempt built 25.00 is recognised as the same work instead of a conflict. Applying a settled key to different work is refused:

python
from ledger_core import OperationMismatch

try:
    operations.post("psp-deposit:evt_88", other_entry)
except OperationMismatch:
    # this key already carries different postings; the derivation is wrong,
    # not the retry
    raise

The fingerprint turns a class of silent corruption into a loud failure at the boundary. It also has a consequence worth knowing before you deploy it, and I will come back to that...

The record has to outlive the process#

Deduplication that lives only in memory is deduplication with a hole in it the width of every restart. A deploy, a crash, an autoscaler killing a pod, and the key that was settled two seconds ago is unknown again, right when the provider is still retrying.

Operations keeps its records as plain frozen values, so you can store them beside the journal and hand them back:

python
records = operations.operations()   # every key settled so far
# ... persist records, restart, reload the journal ...
operations = Operations(journal, records)

restore refuses a record naming entries the journal does not hold, because such a record describes some other journal, and settling a key against a write nobody made is worse than having no record at all. So the record and the entries have to be persisted together, atomically. If they can diverge, you have two truths about the same write, and the whole point of the key was to have one.

The library itself keeps the journal in memory, and persistence stays the host application's job. What the library fixes is the shape: because an Operation is a value with a key, its entry ids, its fingerprint and a timestamp, storing it is a row rather than a serialization problem, and the same key settled in one process stays settled in the next.

What I would do differently#

Two things, both learned the annoying way.

Derive the key at the edge, from something the upstream already owns. The provider's event id, the payout request id, the settlement batch id. Do not generate it inside the retry loop: a key built from uuid4() or from datetime.now() at the top of the handler is a new key on every attempt, which is a dedup mechanism that deduplicates nothing while looking like it works.

Capture the entry the same way. The fingerprint covers entry_id and occurred_at, which means the retry has to rebuild the same entry, not an equivalent one. A handler that stamps datetime.now(timezone.utc) fresh on each attempt produces a different fingerprint and gets an OperationMismatch where it expected its original postings back, technically the system protecting itself, practically a page at three in the morning. Decide the timestamp and the entry id once, when the request first arrives, and carry them through every retry alongside the key. Deterministic ids and a captured timestamp cost nothing at write time and remove a whole category of incident.

The underlying move is small and it probably generalises past ledgers: when a write can be asked for twice, make the second answer be the first result. Refusing the duplicate protects your data and leaves the caller guessing. Returning the original protects your data and ends the conversation. Only one of those keeps the guessing out of six different call sites.

The code is at https://github.com/polycratia/ledger-core.

react

$ new-project --brief

or email hey@polycratia.com