Signing a transaction is the easy part of a custodial wallet. Deciding when the money actually left is where the system breaks.
I have built custodial BTC and ETH wallets and on-chain payment flows, and the recurring failure is not cryptography. It is that you now run two ledgers with different notions of finality. Yours is transactional and instant. The chain's is probabilistic, reorganisable, and occasionally silent for hours.
If a user balance is derived from chain state, every confirmation-depth choice becomes a product decision made by an engineer at 2am. If it is derived from your own ledger, you have to explain every gap between what you recorded and what the chain settled.
What I do now: the internal ledger owns the balance, and the chain is treated as an external settlement source that gets reconciled against it — same way I would treat a bank's statement in a fiat payment system. A withdrawal moves through intent, broadcast, and confirmed as separate recorded states, never one boolean. Stuck and replaced transactions then become normal cases with a place to live, not incidents.
The useful reframing: on-chain is not your database. It is a counterparty you reconcile with.
For those running custodial flows — do you let confirmation depth vary by amount, or keep one fixed rule and eat the tail risk?