The only safe failure state for an outbound payout is unknown

2026-08-20 · 8 min read

Sending money out of a system is not the mirror image of taking it in. An inbound payment can be re-parsed, replayed and reconciled at leisure; an outbound payout that leaves twice is a loss somebody has to chase by hand. The hard part is not the retry logic. The hard part is deciding, after a request that told you nothing, whether you are allowed to try again at all.

Every outbound rail I have worked with has the same shape of failure: bank transfers, card refunds, stablecoin sends from a custodial wallet. You post a request, the connection drops or the gateway answers 502, and the response never arrives. Locally that is an exception. On the provider side the transfer may already be queued, signed or broadcast. If the code maps that exception to failed, and the retry worker treats failed as try again, you have built a machine that occasionally pays twice.

The constraint: the outcome is three-valued, your HTTP client is not#

A payout request has three possible outcomes, not two:

• accepted — the provider took responsibility for the transfer;
• rejected — the provider explicitly refused, with a reason code;
• unknown — you have no idea, and no local evidence will ever tell you.

Almost every HTTP client and most ORM-shaped job runners give you two: it returned, or it raised. The collapse of unknown into the raised branch is the entire bug. Everything below exists to keep those three states apart from the moment the row is written until the moment a human or a reconciliation file resolves it.

There is a second constraint that people discover late. Provider idempotency keys are real, but they are bounded — a key is honoured for some retention window, and after that the same key is just a new request. So you cannot lean on the provider alone. You need your own stable handle, and you need to know how old an in-flight payout is before you touch it again.

Write the intent before you touch the network#

Nothing is sent until a committed row says it is about to be sent. That row is the only thing that survives a process being killed at the wrong moment.

create table payout (
    id              uuid primary key,
    account_id      uuid not null,
    amount_minor    bigint not null check (amount_minor > 0),
    currency        char(3) not null,
    destination     jsonb not null,
    status          text not null default 'pending'
                    check (status in ('pending','in_flight','settled','rejected','review')),
    provider_ref    text,
    attempt_count   int not null default 0,
    in_flight_since timestamptz,
    created_at      timestamptz not null default now(),
    updated_at      timestamptz not null default now()
);

create index payout_stuck_idx on payout (in_flight_since)
    where status = 'in_flight';

create table payout_attempt (
    id              bigserial primary key,
    payout_id       uuid not null references payout(id),
    idempotency_key text not null,
    started_at      timestamptz not null default now(),
    outcome         text check (outcome in ('accepted','rejected','unknown')),
    provider_code   text,
    raw             jsonb
);

Two details in that schema carry most of the weight.

rejected is terminal and unknown is not a status at all — it is in_flight that has aged past its deadline. There is no way to write a payout off as failed without a provider code, because there is no code path that produces one.

The attempts table is append-only. The payout row tells you where the money is; the attempts tell you what you did to it. When something goes wrong at three in the morning, the second question is always the one you cannot answer from a mutated status column.

The idempotency key is the payout id, and nothing else#

This is the mistake I would most like to un-make. It is tempting to derive the key from the business fields — account, amount, currency, day — because it looks self-describing and it deduplicates careless callers for free.

It also silently changes when your serialization changes. Move an amount from a decimal string to minor units, normalise a destination address, reorder a JSON payload before hashing, and the key you send after the deploy is not the key you sent before it. The provider sees a brand new request and does the honest thing: it sends the money again. The diff that caused it looks like a formatting cleanup.

So the key is the primary key of the payout row: an opaque identifier, generated once, immutable by construction, meaningless to anyone but the two systems that share it. Deduplication of careless callers is a separate problem, solved by a unique constraint on the caller's own reference at the point of creation.

Claim, call, and classify the answer rather than the exception#

The worker claims a row and commits that claim before opening a socket.

CLAIM = '''
update payout
   set status = 'in_flight',
       attempt_count = attempt_count + 1,
       in_flight_since = now(),
       updated_at = now()
 where id = (
     select id from payout
      where status = 'pending'
      order by created_at
        for update skip locked
      limit 1
 )
returning id, amount_minor, currency, destination
'''


def run_one(db, provider):
    with db.transaction() as tx:
        row = tx.fetchone(CLAIM)
    if row is None:
        return False
    # The claim is committed. A crash from here on leaves the payout
    # in_flight with no answer, which is exactly the state we want it in.
    submit(db, provider, row)
    return True


def submit(db, provider, row):
    key = str(row['id'])
    try:
        resp = provider.create_transfer(
            idempotency_key=key,
            amount_minor=row['amount_minor'],
            currency=row['currency'],
            destination=row['destination'],
            timeout=20,
        )
    except (Timeout, ConnectionError, ProviderUnavailable):
        record_attempt(db, row['id'], key, outcome='unknown')
        return  # status stays in_flight, deliberately

    if resp.declined:
        record_attempt(db, row['id'], key, outcome='rejected', code=resp.code)
        set_status(db, row['id'], 'rejected')
        return

    record_attempt(db, row['id'], key, outcome='accepted', code=resp.code)
    set_provider_ref(db, row['id'], resp.id)  # still in_flight

The classification table is short enough to hold in your head, and worth writing down next to the code:

Note that accepted is not settled. The provider agreeing to move money is not the money having moved. Settlement arrives later, from a webhook or a reconciliation file, and it is the only thing allowed to write settled.

Resolve by asking, never by guessing#

Everything in in_flight past a deadline belongs to a resolver whose only move is to ask the provider what it knows, keyed by the handle you already own.

def resolve_stuck(db, provider, deadline=timedelta(minutes=5)):
    for row in db.fetch(STUCK, deadline):
        key = str(row['id'])
        found = provider.find_transfer(idempotency_key=key)

        if found is None:
            # The provider has no record. Re-queuing is safe only because the
            # key is unchanged and still inside the provider's retention window.
            if age(row) < provider.key_retention:
                set_status(db, row['id'], 'pending')
            else:
                set_status(db, row['id'], 'review')
            continue

        if found.state in TERMINAL_OK:
            set_status(db, row['id'], 'settled', provider_ref=found.id)
        elif found.state in TERMINAL_BAD:
            set_status(db, row['id'], 'rejected', code=found.code)
        # anything else is still in motion: leave it alone

The branch that matters is the one that does nothing. A transfer the provider still considers pending is not stuck, it is slow, and the only correct action is to wait. Most double-send incidents I have looked at come from a resolver that could not sit still.

What I would do differently#

Make attempts append-only on day one. I have shipped versions that only mutated a status column, and every incident afterwards started with reconstructing what the worker actually did from log fragments.

Alert on the age of the oldest in-flight payout, not on error rate. Error rate is noisy and mostly harmless. A payout that has been in flight far longer than the rail's normal settlement time is the signal, and it fires even when nothing is throwing.

Require a provider code for every rejection. A rejected row with a null code means some code path invented a terminal state, and that path is a bug worth failing loudly on.

Give the review queue an action that resolves without sending. When a human confirms externally that money moved, they need to record a provider reference and close the payout. If the only available button re-submits, someone will eventually press it.

The design is really one rule with consequences attached: an outbound payout has three outcomes, and the third one is not an error. Once unknown has a home in the schema, a deadline, and a worker whose job is to ask rather than to act, the rest of the state machine mostly writes itself.