A request key stops a retry, not a second client

2026-10-31 · 9 min read

An idempotency key answers exactly one question: is this call a repeat of an earlier call under the same key? It says nothing about a different client submitting the same withdrawal under a key you have never seen, and that is the case that pays your user twice. Closing it takes a second index (the withdrawal id mapped to the key that claimed it) and a refusal that names the holder instead of just saying no.

I have been shipping payment and crypto rails since 2018, and the duplicate-payout postmortems I remember were almost never a client retrying badly: they were two callers who each behaved correctly, in isolation. The code below is from the withdrawals package I keep at github.com/polycratia/withdrawals, where the submission ledger is small enough to read in one sitting.

The question a key can answer#

Submissions.submit(key, request, send) keeps a dictionary _records from request key to a _Record. The record holds a _fingerprint of the withdrawal (the tuple of id, amount, currency and destination) and an outcome that starts as None. Every decision it makes starts the same way:

def _claim(
    self, key: str, fingerprint: _Fingerprint, request_id: str
) -> WithdrawalRequest | None:
    with self._lock:
        record = self._records.get(key)
        if record is None:
            self._claim_id(key, request_id)
            self._records[key] = _Record(fingerprint)
            return None
        if record.fingerprint != fingerprint:
            raise IdempotencyConflict(
                f"request key {key} belongs to another withdrawal"
            )
        if record.outcome is None:
            raise SubmissionInFlight(
                f"request key {key} is claimed and its outcome is not known yet"
            )
        outcome = record.outcome
    logger.info("submission %s replayed for withdrawal %s", key, request_id)
    return outcome

Three outcomes, all of them useful. A known key carrying a different fingerprint is a client reusing a key for another withdrawal, so that is a conflict rather than a silent overwrite. A known key with no outcome yet is in flight. A known key that already produced an outcome replays it, and submit returns without calling send at all.

Now read the first line again. self._records.get(key). Every branch in that function is scoped to the key the caller brought. A second caller who brings a fresh key matches nothing, falls into the record is None arm, and runs the send. The whole idempotency story is a cooperation agreement with one client, and it holds exactly as long as the client is one client.

In production that assumption breaks in dull ways. A user taps withdraw on a phone and then on a laptop after the first request times out behind a load balancer. An operator re-drives a stuck queue item, and the redrive path mints its own key because the original was never persisted anywhere the operator can see. A provider webhook gets replayed and the handler reads it as a new instruction. None of these is a retry in the sense the key understands. All of them are a second payout.

The second index#

So the ledger keeps a second dictionary, _claims, from withdrawal id to the request key that claimed it, and takes the claim in the same lock-held block that creates the record, before the send runs rather than after it returns:

def _claim_id(self, key: str, withdrawal_id: str) -> None:
    holder = self._claims.get(withdrawal_id)
    if holder is not None and holder != key:
        raise IdempotencyConflict(
            f"withdrawal {withdrawal_id} is claimed by request key {holder}"
        )
    self._claims[withdrawal_id] = key

Four lines, and they change what the ledger is. _records makes a key idempotent. _claims makes a withdrawal exclusive. The first protects the honest client, the second saves you from a caller you did not design for.

The detail I care about is that the refusal names the holder. A bare conflict leaves the second caller with nothing to do but guess, and a caller with nothing to do will usually retry. Naming req-1 turns a dead end into a lookup: claimed_by(withdrawal_id) gives the key, outcome(key) gives what that key produced, and the second caller can report the real state of the withdrawal instead of creating a new one.

def test_a_second_key_for_a_claimed_withdrawal_is_a_conflict():
    ledger = Submissions()
    sender = Sender()
    ledger.submit("req-1", make(), sender)

    with pytest.raises(IdempotencyConflict) as conflict:
        ledger.submit("req-2", make(), sender)

    assert "req-1" in str(conflict.value)
    assert sender.calls == 1
    assert ledger.outcome("req-2") is None

Two of those assertions carry the whole mechanism. sender.calls == 1 says the send never ran for the loser. ledger.outcome("req-2") is None says the loser left no trace: a refused claim does not half-register a key that a later caller could mistake for a successful submission.

The same guard sits on the write path. resolve(key, outcome) goes through _claim_id when no record exists for the key, so a callback arriving under a fresh key cannot staple an outcome onto a withdrawal that another key already holds. Without that, the claim would be a lock on the front door with the back door propped open: the second caller could not send, but could still declare what happened.

The claim has to survive a send that raised#

The harder decision is what the ledger does when send throws. It does almost nothing, on purpose:

try:
    outcome = request if send is None else send(request)
except BaseException:
    logger.warning(
        "submission %s left withdrawal %s with an unknown outcome; "
        "the key stays claimed until it is resolved",
        key,
        request.id,
    )
    raise

It logs, and re-raises with the key still claimed and its outcome still None. So the retry that follows does not get another attempt: it gets SubmissionInFlight, and a different key gets IdempotencyConflict. The test name says it better than I can. A withdrawal in flight is claimed as firmly as a sent one.

def test_a_send_that_raises_keeps_the_key_claimed():
    ledger = Submissions()
    sender = Sender()

    with pytest.raises(RuntimeError):
        ledger.submit("req-1", make(), boom)

    with pytest.raises(SubmissionInFlight):
        ledger.submit("req-1", make(), sender)

    assert sender.calls == 0
    assert ledger.outcome("req-1") is None

This is the asymmetry people argue with, so it is worth stating plainly. An exception from a rail tells you the call failed. It does not tell you the transfer did not happen. A socket that times out after the request reached the node, a client that dies between broadcast and the response, a provider returning a 500 from behind its own queue: in all of those the money may already be moving. On an external rail the second send is a second transaction, with its own fee, and nothing in the protocol will give it back. A ledger that releases the key on exception has decided to guess, and it guesses in the direction that costs real money.

The case where you do know nothing left the account is different and has its own machinery. A rail that refuses the transaction cleanly raises BroadcastRejected, and broadcast wraps the send so that the refusal releases the hold in the same call rather than stranding the reserved funds. Note what that does and does not cover. It gives the funds back; the request key is a separate question, and it stays claimed until somebody closes it.

Closing an in-flight key by hand#

There are exactly two ways out, and they differ in what the caller is asserting.

resolve(key, outcome) is the "I looked, here is what happened" exit. It fingerprints the outcome, which matches the request's fingerprint because a transition changes a withdrawal's state and not its identity: id, amount, currency and destination are all still the same. So an operator who finds the transaction can hand the ledger the real request, and the key closes around it:

def test_a_claimed_key_closes_once_the_truth_is_known():
    ledger = Submissions()
    sender = Sender()
    with pytest.raises(RuntimeError):
        ledger.submit("req-1", make(), boom)

    found = make().approve().start_sending().mark_sent("0xdeadbeef")
    ledger.resolve("req-1", found)

    assert ledger.submit("req-1", make(), sender) is found
    assert sender.calls == 0

release(key) is the "I looked, nothing was sent" exit, and it is guarded on both sides:

def release(self, key: str) -> None:
    key = _require_key(key)
    with self._lock:
        record = self._records.get(key)
        if record is not None:
            if record.outcome is not None:
                raise IdempotencyConflict(
                    f"request key {key} already produced an outcome"
                )
            withdrawal_id = record.fingerprint[0]
            if self._claims.get(withdrawal_id) == key:
                del self._claims[withdrawal_id]
        self._records.pop(key, None)

A key that produced an outcome cannot be released at all, since that would re-open a withdrawal which already has an answer. And the id claim is dropped only when it still points at this key, so a release arriving late cannot evict a claim that has since moved to somebody else. After a release the key is free to run the send again, and so is a different key: the withdrawal id goes back to unclaimed, and the next caller to arrive becomes its holder.

That is the price of the design, and it is an operational price rather than a code one. Keys left in flight by a raised send do not expire. Someone has to go and look at the rail. I have not found a way to automate that honestly: every automatic release is a timer deciding, with no evidence, that a rail which stopped answering also stopped working.

What I would do differently#

Both indexes sit in one process behind one threading.Lock, which is right for what this package is and is also a single-process guarantee. The shape transfers to a deployment, the storage does not. Across several API workers the claim has to be a uniqueness constraint where the rows actually live, so that two concurrent submissions race into one database and get one winner, rather than racing into two dictionaries and getting two.

I would also be stricter, earlier, about where the withdrawal id comes from. The claim is only as strong as the id's uniqueness: if a client mints the id as well as the key, a second caller simply mints a second id, _claim_id sees no holder, and the exclusivity you thought you had evaporates. The id has to be assigned when the request is created on your side, and the submission ledger has to be the only path to a send.

The useful way to hold all of this, I think, is that a request key and a withdrawal claim answer different questions. One is about calls, the other about money. Systems that conflate them tend to look correct in every test where the client cooperates.