Everyone building a custodial wallet worries about key storage. The part that actually breaks is the withdrawal queue.
Key management is a solved problem with known answers. Withdrawals are not, because they span two systems that cannot share a transaction. Your ledger debit is reversible. The broadcast is not. Between those two moments sits every incident I have seen in custodial systems since 2018.
A node call times out and you do not know whether the transaction was broadcast. A retry looks safe on the application side and is catastrophic on chain. A user's balance drops before finality and rises again after a reorg. A nonce is reserved by one worker and consumed by another.
The approach that has held up for me: the ledger debit and the intent to send are written in the same transaction, and nothing broadcasts from that path. A separate worker owns broadcast, owns the nonce, and is the only thing allowed to move a withdrawal forward. The chain state is treated as an external fact that gets observed and recorded, never as something the request path can assume. Every state change is append-only, so a stuck withdrawal can be read as a history rather than guessed at.
The useful reframe: a withdrawal is not an action. It is a long-running agreement between your ledger and a chain that does not know your ledger exists.
For those running custody in production — where does your withdrawal path lose certainty first: broadcast ambiguity, nonce contention, or confirmation depth?