The hardest part of a custodial wallet is not key management. It's that a withdrawal is a distributed transaction between your database and a chain that will never roll back for you.
The naive version looks fine in review: open a transaction, debit the user's balance, call the node to broadcast, commit. It survives testing because nothing fails in testing.
In production the node call times out after the transaction is already in the mempool. Now you either roll back and pay out twice, or commit and hope. Neither is an answer.
How I build it instead: the debit and a pending withdrawal record commit together, and nothing touches the chain inside that transaction. A separate worker picks up the pending row, broadcasts, and moves it through a state machine — pending, broadcast, confirmed, failed — keyed by the transaction hash. Broadcasting the same signed payload twice is harmless. Signing a second one is not, so the nonce or UTXO selection has to be owned by one writer.
The reconciler then walks confirmed chain state back against the ledger, and anything that does not match gets flagged rather than auto-corrected.
The rule I keep coming back to: never let an irreversible external effect share a commit boundary with your own state.
For anyone running custodial withdrawals — where does yours get stuck most, stuck broadcasts or the fee bumping?