build log
- $ · permalink
A fast database does not give you a fast API. Query time is the slice of the latency budget you actually control, and most of that budget belongs to systems you don't own.
Look at where a request in a payments or marketplace backend spends its time. A checkout call quotes rates from several carriers. A signup call waits on a KYC provider's decision. A payout call waits on a payment provider that is quick on a good day and tells you nothing on a bad one.
Each of those is a network round trip to an endpoint with its own load and its own queueing, plus an incident you will never be told about.
Serialize three of them and your median is the sum. Your tail is whichever dependency is having a bad afternoon.
Then the blocking spreads. A worker waiting on an external response still holds its slot, and often still holds its database connection. One slow provider quietly turns into a globally slow API, while the database dashboard stays green: which is probably why people keep optimizing the one layer that was never the problem.
The fix is a boundary, and no index will get you one. Decide explicitly what the request path is allowed to wait on. Everything else gets a timeout you chose, or a fallback answer, or a pending state that reconciles asynchronously. Cache what is stable: carrier rates change far more slowly than checkouts happen. And measure latency per dependency instead of as one number, or you average the problem away.
I write up more production trade-offs like this one at https://polycratia.com/c/4cbf07a8
Worth knowing where the time actually went, the last time you chased a slow endpoint...
- $ · permalink
A rewrite is rarely a code problem. It is a problem of everything the code does that nobody wrote down.
Five questions decide it before the first commit.
What behaviour is load-bearing but unintentional? Old systems collect quirks, and downstream consumers have already adapted to them. Rounding that falls one way every time. A field that arrives empty instead of null. You can reproduce it or you can break it, but decide that on purpose rather than find out in production.
Where does state live outside the database? Queue messages in flight, cron runs that are half done, retry counters sitting in a worker. A cutover moves rows. What is mid-flight it hardly ever moves.
Who calls this that you do not control? Integrations you cannot version on your own schedule: their change window sets your timeline, not the other way round.
Can both systems run at once and be compared? If you cannot push real traffic through the old path and the new one and diff the outputs, you are not migrating, you are hoping. Reconciliation is the migration.
What is the unit of rollback? One endpoint, one service, one table. If the honest answer is "the whole thing", your plan has no brakes.
In Python 2 to 3 migrations and monolith-to-services splits the failures cluster in the same place: not the new code, but the assumptions the old code was quietly satisfying for years. The new implementation is usually correct. Correct against a specification nobody ever agreed to...
I wrote more on how I approach cutovers like this: https://polycratia.com/c/ef123070
Of those five, one gets skipped most often in the rewrites you have been part of.
- $ · 9 min readParse crypto amounts like hostile input, declare rounding first
Every crypto amount enters a system as a string: a form field, a JSON body from an exchange API, a row in a payout file, a webhook from a provider. Between that string and the ledger two decisions…
- $ · permalink
An AI agent that works in testing and fails in production usually did not get worse at reasoning. It just met a second copy of itself.
In testing there is one agent, running one task, in one clean sequence. In production that same agent runs concurrently across users, retries itself on a timeout, and gets replayed by whatever queue sits in front of it. Now two runs are touching the same external state at the same time, and that state lives outside your database: in a payment provider, a CRM, a mailbox, a document workflow. You cannot wrap it in a transaction and roll it back.
This is not a new class of bug. It is the same failure mode I have been designing around in payments since 2018: an action that is safe once and destructive twice. The difference is that a payment client retries on a fixed rule you wrote. An agent decides to retry on its own, with slightly different wording, and your dedup key never matches.
What I do about it is boring, and it works. Every side effect an agent can trigger gets an operation record before the call, keyed on the business fact (this user, this invoice, this document) and not on the agent's generated text. The tool checks that record first. If the operation already exists, it returns the prior result instead of acting again. The agent is allowed to be non-deterministic. The effect layer is not.
The second thing: the tool layer refuses, loudly, rather than improvising. A tool that returns an empty result on failure teaches the agent that nothing was there, and the agent moves on. Absence and failure have to be different return values, or the model will confidently reason from a lie.
My take: reliability of an agent is not a property of the model. It is a property of the boundary you put between the model and anything irreversible. Test suites almost never exercise that boundary, because the boundary only breaks under concurrency and partial failure, which is exactly what staging does not have.
I have written more on designing those effect boundaries here: https://polycratia.com/c/6c4a27b7
If you run agents against real external systems, something already forced your tool layer to be idempotent: duplicate writes, duplicate charges, or probably a duplicate message that went out to a customer...
- $ · 7 min readIssuing deposit addresses without holding a private key
- $ · permalink
That endpoint was fine for almost everyone. It was slow only for the accounts carrying the most data.
A 3-second response usually gets read as a code problem: profile the handler, add an index, move on.
But the median looked fine. So did the 90th percentile. The slowness sat in a thin tail, and everyone in that tail had one thing in common: unusually large related data. On a lending deal that meant many investors sharing one loan, each with a position to resolve. In a catalog it meant one product with hundreds of variants.
The handler did a fixed amount of work per related row. Cheap at five rows. Ruinous at five hundred. Latency was never a property of the code, it was a function of the data shape the caller happened to have.
Which is why dashboards missed it. They average across users whose data does not look alike, and the shape that breaks you is a minority by definition.
So now I log cardinality next to duration: how many rows, children or positions a response actually assembled. Then I plot latency against size instead of against time. A flat line means fixed cost. A slope means per-row work you will meet again, larger, later.
The slope is the bug. The seconds are only the symptom.
The uncomfortable part is that this failure mode scales with success. Your heaviest, longest-tenured, most valuable accounts hit it first, and they are the ones least likely to file a support ticket about it.
I wrote up more of these slow-response post-mortems, including the ones that hid the longest, here: https://polycratia.com/c/682ba0de
For you it might have been a specific tenant, or a payload size, or a dimension you were not logging at all...
- $ · 7 min readERC-20 amounts are meaningless without the token's decimals
An ERC-20 balance is a uint256 counting the token's smallest unit, and nothing in that integer tells you how many units make one token. You have to ask the contract. Assume eighteen decimals against…
- $ · permalink
Every dependency in the request was healthy. The endpoint was still slow.
Both of those were true at the same time.
This is where a lot of latency work goes wrong. You open the database dashboard: queries are fine. You open each downstream service, and each one reports a reasonable average. Nothing looks broken, and users still wait.
A user request does not experience averages. It experiences the slowest thing on its own critical path.
I hit this over and over on shipping and checkout endpoints in cross-border commerce. One call would read the catalog, then fan out to several carriers for live rates, then touch payment and currency logic before it could answer anything. Each integration was fine on its own. But once a single request depends on six things, the chance that at least one of them is having a slow moment stops being an edge case. The tail latency of that endpoint ends up much closer to the maximum of its calls than to any average you happen to be staring at.
So I stopped measuring components and started measuring the request. One timeline per request, every external call on it, ordered, waiting time visible. That view usually answers the question in minutes, and what it turns up is almost always something nobody suspected: a serial chain of calls that could have run in parallel, or a retry policy quietly tripling one leg, or an enrichment step nobody needs before the response goes out.
The fixes are unglamorous. Give every external call an explicit time budget instead of the default timeout it inherited from some library. Run independent calls concurrently instead of chaining them. And decide up front what the response looks like when a dependency does not answer in time, because a degraded answer (a cached rate, a narrower list) beats holding a request hostage to the slowest participant.
Latency is a property of the composition, not of the parts. That is why a fast database and a slow API are not a contradiction.
I write up more of these production trade-offs here: https://polycratia.com/c/7234d751
The last time you chased a slow endpoint, the time was probably hiding somewhere you weren't looking...
- $ · 7 min readIdempotency belongs in the ledger, not in every caller
A payment callback that arrives twice must not move money twice. The usual answer is a unique constraint on the entry id, which turns the second attempt into an error, and an error is not the same…
- $ · permalink
Most KYC integrations don't fail on the happy path. They fail on the states nobody modelled.
Teams treat verification as a gate: the user submits documents, the provider answers, you set verified = true and move on. That works until production.
Then you meet the middle. A provider returns pending and never calls back. A document expires while the account is live. A manual reviewer wants more information. A user who passed six months ago trips a sanctions re-screen. None of that is approved or rejected, and a boolean has nowhere to put it.
What I do instead: model verification as its own state machine with explicit intermediate states, each one persisted with the reason it was entered and who or what can move it forward. Provider responses are events applied to that state, not the state itself. Re-verification is a normal transition, not an exception.
The practical payoff is the moderation queue. A compliance reviewer opens a case and sees the current state plus the justification for it, instead of reconstructing history from provider logs. And when a user is stuck, you can answer why in one query.
The other benefit: onramps and payouts get to ask one question, is this user permitted to do this action right now, rather than every service inventing its own reading of a flag.
If you run KYC in production, there's probably one intermediate state that cost you real money before you gave it a name...
- $ · 8 min readunder_investigation is a status, not an excuse
A scanner tells you a component in your build carries a known advisory. It cannot tell you whether the vulnerable code is reachable in your product, and that judgement is the entire content of a VEX…
- $ · permalink
not_affected is a claim you make on someone else's behalf. under_investigation is usually the honest answer.
The failure mode is quiet. A finding lands, nobody can prove the vulnerable path is actually reachable, and it gets closed as not_affected because that clears the board. Unknown reachability just became a negative assertion. The backlog didn't shrink — it turned into assurance a downstream consumer will rely on.
I have hit this exact shape in payment attribution. Funds arrive, they don't match cleanly, and the tempting move is to guess the owner so the exception queue stays empty. The discipline that holds is the opposite: never guess, flag the unmatched, and treat "not resolved yet" as a real state with its own review path — not a gap you paper over.
Same rule for VEX. under_investigation is a status, not an excuse. And any status that survives review has to carry its why, because a verdict without a justification can't be re-checked when the code moves underneath it.
Where does your triage quietly convert unknown into no?
https://polycratia.com/c/39307b9c
- $ · 8 min readThe ERC-20 that returns nothing still moved your money
Three behaviours in deployed token contracts break integrations that trust the return value of a transfer: contracts that return no data at all, contracts that refuse to raise an allowance that is…
- $ · permalink
Most money bugs I have chased were not float bugs. They were rounding decisions nobody made.
The usual story: an amount shows up as a string from an API, a webhook, a user form. Somebody parses it into whatever numeric type is nearest to hand. Then it goes through a conversion, a fee, a split across investors, a display layer. Every one of those steps rounds, and none of them was asked to. The final number is whatever the last operation happened to leave behind, and three weeks later during reconciliation it is off by a unit nobody can explain.
So I treat inbound amounts the way I treat any hostile input.
Parse from the string, never from a float. The string is what the counterparty actually sent; a float is already a lossy reading of it.
Validate against the asset's own precision. Eighteen decimals is normal on one chain and nonsense on a card rail. If a value carries more digits than the asset can represent, you do not quietly truncate it, you reject it at the boundary. Truncating means you have silently accepted an obligation you cannot settle exactly.
And make rounding a declared policy, checked before the arithmetic runs, not a side effect you find afterwards. Half-up, floor, banker's: the specific choice matters far less than the fact that it was chosen explicitly, is visible in the code, and is the same on the ledger side as on the display side.
That is the thinking behind cryptomoney, a small open-source library I built for exactly this: precision-aware parsing, explicit rounding policy, and loud failure on anything that does not fit the asset.
The deeper point is architectural. Rounding is a business rule. Leave it implicit and it does not disappear, it gets spread across every function that touches an amount, and each one gets a vote. Ledger drift is rarely one dramatic bug. It is a hundred small unowned decisions agreeing to disagree...
When an amount crosses a boundary (chain, rail, service, currency) precision and rounding are part of its type, not a formatting concern for later.
I wrote up more of the design reasoning on the blog: https://polycratia.com/c/36032cb1
If you run multi-asset ledgers, you enforce precision somewhere: at ingestion, at the type, or at the database column. I have seen all three, and the failure modes are very different.
- $ · permalink
A token transfer that returns nothing can still have moved your money. And a transfer that returns true can still have moved less than you asked for.
ERC-20 is a standard the way a handshake is a standard. Plenty of deployed contracts leave out the boolean return, so a strict integration reverts on a transfer that actually went through. Some refuse an allowance increase unless you set it to zero first, so your approve call fails against a balance that is sitting right there. Some take a fee out of the transfer, so the amount you credited internally is bigger than the amount that arrived.
Each one breaks a different layer. The first breaks your call site. The second breaks your onboarding flow. The third breaks your ledger quietly, and that is the one you find weeks later during reconciliation.
The habit that survives all three: don't trust the return value as evidence of the effect. Read the recipient balance before, read it after, credit the difference. The transfer is a request, the balance delta is the fact.
I wrote it up properly here: https://polycratia.com/c/047ca8ee
The missing return, the approve reset, the fee on transfer: you have probably met at least one of them in production...
- $ · 8 min readA withdrawal is a queue entry before it is a transfer
Most withdrawal code collapses two different events into one function call: the user asking for money to leave, and the money actually leaving. Once those are the same operation, everything you…
- $ · permalink
The service that hands out deposit addresses should be incapable of spending from them.
Not "trusted not to". Incapable.
That is the whole reason to derive deposit addresses from an account level extended public key. You hand the address service an xpub and an index. It derives the next address and gives it to a user. It holds no private key, so there is nothing on that box for an attacker to steal and move funds with. Signing sits somewhere else entirely, on a different trust boundary, usually touched by a different process and different people.
I have built custodial wallets for BTC and ETH, and this is the design decision that ages best. Most other key handling policy I have seen is a promise you keep by discipline. This one is kept by arithmetic: public derivation cannot produce the private half.
What surprises people is where the risk goes once you do this.
It goes into bookkeeping. Two things now decide whether a deposit is ever seen: the derivation path and the gap limit.
The derivation path is your account structure. Get it wrong once, in a migration or a rewrite, and you end up watching a perfectly valid branch of a tree that no user was ever given an address from. Funds land at addresses nobody is looking at. The chain is fine. Your ledger says nothing.
The gap limit is worse, because it fails quietly and only under real usage. You scan a window of unused addresses ahead of the last one you saw money on. Hand out addresses faster than deposits arrive (bots, abandoned checkouts, users who generate and never fund) and the funded address drifts past the end of that window. The deposit is confirmed on chain and invisible to you. Nobody gets an error. Support just gets a message saying "I sent it, where is it".
So I treat the watch set as a first class piece of state, not a derived convenience: every address handed out is recorded at issue time with its full path, and the watcher runs off that record rather than an optimistic scan window. The gap limit becomes a fallback for recovery, not the source of truth.
This came out of chain-addresses, a small open source project of mine for deriving and tracking deposit addresses. The longer write up on derivation paths and gap limit failure modes is on my blog: https://polycratia.com/c/d5daf33e
If you run custodial deposits in production: do you drive your watcher from issued addresses, or from a scan window over the xpub?
- $ · permalink
The fastest way to lose control of a custodial wallet, I think, is to model a withdrawal as a transfer.
A transfer is a single moment. It either happened on-chain or it didn't. If that is your model, then the instant a user hits withdraw you are already committed. There is no safe point to cancel. No window to batch several outputs into one transaction either, and no chance to re-price the fee when the network moves under you.
A withdrawal request is not a moment. It is a record with a lifecycle: requested, held for review, approved, queued, signed, broadcast, confirmed. Every one of those is a place where a human can still step in, or a risk rule, or a fee policy.
I have built custodial BTC and ETH wallets and on-chain payment rails, and the pattern holds every time. Once the request is its own row with an explicit state machine, broadcasting becomes one transition inside the operation rather than the entire operation. Cancellation becomes a state change instead of an apology. Batching becomes a scheduling decision. Compliance holds stop being special-cased hacks bolted onto the send path.
Infer the state from the chain instead, and you get a system that can only tell you what already happened...
Wrote up the full argument here: https://polycratia.com/c/4298b9da
If you run withdrawals in production, I'd like to know what forced you to add the request record: a cancellation you couldn't honour, or probably a fee spike you couldn't re-price.
- $ · 8 min readA ledger should not own a money type
Most ledger libraries ship their own Money class, so any system that already had a money type ends up with two of them and a conversion layer in between. A ledger does not need to own a money type.…
- $ · permalink
The moment a ledger ships its own Money class, your system has two money types.
One from your currency library. One from the ledger. Every posting becomes a conversion at the boundary, and every conversion is a place where scale, rounding, or currency can quietly change.
In the payment systems I have built — per-user account ledgering, attribution by requisites, reconciliation against what the bank actually settled — the bugs that take longest to find are almost never in the arithmetic. They live in the translation layer nobody thinks of as logic.
So I stopped letting the ledger define money. A ledger needs exactly three things from a value: an exact amount, a currency, and arithmetic that obeys the rules of that currency. That is a protocol, not a class. Anything that satisfies it can be posted.
One money type per system, owned by the currency library, consumed by the ledger.
The trade-off is honest: you give up having the ledger validate money for you, and you take on making your own type correct. I would rather get one type right than keep two in sync forever.
Where does your system convert between money representations today — and do you know every place it happens?
https://polycratia.com/c/c9290dd5
- $ · permalink
The chain can tell you a transaction confirmed. What it can't tell you: why someone asked for it, who approved it, or whether it should have gone out at all.
That gap is why I model a withdrawal as its own record before it is a transfer.
Requested, approved, signed, broadcast, confirmed: explicit states, explicit transitions, each one written down when it happens, not reconstructed later by scanning the chain.
The payoff is everything that happens before broadcast. While a withdrawal is still a request, you can cancel it, hold it for review, reprice it, or batch it with others heading to the same place. After broadcast none of that exists. You have one irreversible fact and a support queue.
Inferring state from the chain also erases the transitions that matter for audit. A confirmed transfer looks identical whether a human approved it, it was auto-approved under a limit, or it was retried after a stuck broadcast. Three different events, three different consequences, and only the record keeps them apart.
I've rebuilt this shape enough times across custodial wallets, exchange payouts and stablecoin rails that I turned it into an open-source project. I called it withdrawals, because the state machine barely changes between them.
Longer write-ups on payments and crypto rails live here: https://polycratia.com/c/60317e09
If you run custodial payouts, I think the interesting part is where you draw the auto-approval line: amount, destination history, or something else...
- $ · 9 min readFee estimation belongs inside the coin selection loop
Coin selection on a UTXO chain is usually written as two steps: pick inputs until they cover the amount, then compute the fee and adjust. That order cannot be made to work. The fee is a function of…
- $ · permalink
The fee for a Bitcoin withdrawal depends on inputs you haven't chosen yet.
That sentence sounds like a paradox, and in coin selection it is a genuine loop. The fee is a function of transaction size. Size is a function of how many inputs you spend. And how many inputs you need to spend depends on the fee, because the fee comes out of the same funds.
The usual implementation pretends the loop isn't there. Select inputs to cover the amount, then bolt the fee on afterwards. Now the selected set no longer covers amount plus fee, so you pull in one more input, which makes the transaction larger, which raises the fee again. Or the code quietly resolves it by shrinking the change output until it drops below dust — and that change is not returned to the user, it's handed to miners.
When I built custodial BTC wallets, the fix was structural, not arithmetic: every candidate input set is evaluated with its own fee at the current rate, including the marginal cost of the change output, and selection only terminates when a set covers amount plus its own fee.
Fee estimation belongs inside the selection loop, never after it.
If you run on-chain payouts: does your selector price the change output, or discover it later?
https://polycratia.com/c/43729555
- $ · permalink
Every KYC integration I've seen treats the vendor decision as the hard part. It isn't. The hard part is what happens when the vendor says "maybe".
Automated verification gives you three outcomes, not two: pass, fail, and needs a human. That third bucket is where the system actually lives. On a fiat-to-crypto onramp I built, the review queue was a first-class service — not a spreadsheet, not an admin page bolted on at the end.
Because a queue has properties nobody plans for. A user sits in limbo with money already in flight. Two moderators open the same case. Someone approves an applicant whose documents expired while they waited. A retry from the vendor arrives after a human already decided, and now you have two verdicts for one identity.
So I design the moderation path the way I'd design a payment path: explicit states, one owner per case, decisions written as immutable events rather than a status column someone overwrites. The audit trail isn't a compliance checkbox — it's the only way to answer "why is this account open" six months later.
What I'd tell anyone starting: build the human-review flow in the first sprint, not the last. It determines your onboarding latency more than the vendor's SLA does.
If you run KYC in production — what's your actual bottleneck: vendor response time, or the queue behind it?
- $ · 7 min readExactly-once deposits: a broker cannot deduplicate a reorg
A service that credits user balances from on-chain deposits has to tell its consumer about each deposit exactly once. The usual reflex is to push the problem onto the transport: turn on…
- $ · permalink
Hardcoding 18 decimals is a bug that only surfaces in production, and only with real money.
ERC-20 does not fix precision. Each contract reports its own decimals value, and the client is expected to ask. Most tokens answer 18. Several of the ones people actually move — USDT among them — answer 6. Same interface, different arithmetic.
Get that wrong and nothing throws. The transfer encodes cleanly, the transaction succeeds, and the amount is off by twelve orders of magnitude in one direction or the other. On-chain that is final.
So I stopped treating precision as a constant. In erc20-transfers, an open-source project I maintain, decimals is read from the token contract and cached per token per chain — never assumed, never inherited from a config file someone copied between environments.
Conversion happens once, at the edge. Human-readable decimal in, integer base units through the whole system, human-readable out. No float touches the middle.
The part that took longer to accept: the encoder refuses a value the token cannot represent. Ask it to send an amount with more precision than the token has, and it errors instead of truncating. Rounding money is a business decision. A transfer library is the wrong place to make it silently — a rounded-down remainder that no one chose becomes a reconciliation ticket weeks later, when the only evidence left is a hash.
Most integration bugs I have hit in crypto payments are not cryptography. They are unit conversion wearing a serious hat.
I keep the longer write-ups on my blog, if this is the kind of thing you deal with: https://polycratia.com/c/60418fe5
For those running token transfers in production — where do you draw the line on rounding? Reject at the edge, or accept and record the remainder somewhere explicit?
- $ · permalink
Every team I've seen chase exactly-once deposit notifications went looking in the queue. The answer was in the watcher's database.
Here is the shape of the problem. A watcher polls the chain, sees a deposit, publishes a notification. Then it polls again and sees the same deposit. Then it restarts mid-batch and re-scans the block range it already processed. Then a reorg rewrites the block it read. None of that is the queue's business — by the time a message exists, the duplicate has already been created upstream.
So the decision moves earlier. Before publishing anything, the watcher writes a row keyed by the thing that is actually unique on-chain, and lets the database reject the second attempt. Publish only if that write is new. Restart-safe, poll-safe, and it survives a consumer that resubscribes from an older offset.
My take: delivery guarantees move messages, they do not define identity. Identity is yours to persist. Once the watcher owns it, the consumer's queue semantics stop mattering — at-least-once delivery becomes acceptable, because the duplicate never entered the stream in the first place.
I wrote up the full reasoning here: https://polycratia.com/c/09828aa7
If you run a chain watcher: what is your dedup key — transaction hash, hash plus output index, or something you derived yourself?
- $ · 9 min readA deposit is confirmed by the tip, not by first sight
Crediting on-chain deposits fails in a specific way. The watcher sees a transfer, writes a row, and starts incrementing a confirmations column on a timer. Two things are already wrong: the count is…
- $ · permalink
Most KYC integrations fail not at the provider API, but at the moment a human says "actually, look again".
The provider side is the easy part. You send documents, you get a decision, you store it. What breaks is everything downstream of that decision.
Because verification is not a boolean. It is a state that moves — sometimes backwards. A user gets approved, then a document expires. A compliance officer reopens a case after a rescreen. A borderline profile sits in manual review while the user is already trying to move money.
I have built onramps where KYC moderation happened in real time, with a compliance team working a live queue. What made those systems survivable was refusing to collapse verification into a single flag on the user row.
Instead: a case with its own lifecycle, an append-only trail of who decided what and on which evidence, and a separate question the payment path asks — is this user allowed to do this specific action right now. Not "is_verified".
The difference shows up the first time you have to explain to a regulator, or to your own ops team, why a transaction was permitted eight months ago.
If you run KYC in production: what forced you to redesign first — re-verification, or the audit trail?
- $ · permalink
A deposit with six confirmations right now can have two confirmations an hour from now.
That single fact breaks most first attempts at a deposit watcher, because confirmations get stored as a counter and incremented once per poll. They are not a counter. They are a value derived from the current chain tip, and the tip moves — occasionally backwards. Increment it and you end up crediting a balance against a block that no longer exists.
The second half of the problem is the one users actually see. Across polls, restarts and re-scans, the same first inclusion surfaces again and again. First inclusion is an event. Exactly-once notification is the product: one message per deposit, per user, however many times the watcher happens to observe it.
So I keep them as two separate problems with two separate fixes. Recompute depth against the tip on every pass instead of trusting stored state. Key the notification on the deposit itself, never on the observation that produced it — idempotency belongs at the notification boundary, not inside the poll loop.
Where does your watcher break first: reorg handling, or duplicate notifications after a restart?
https://polycratia.com/c/e55a4bac
- $ · 9 min readCoin selection should return a reason, not None
A withdrawal fails at three in the morning and the log line says the coin selector returned None. That one value covers at least four different situations: two of them are answered by funding the…
- $ · permalink
Confirmations are not a counter you increment. They are a value you recompute.
That distinction decides whether a deposit system survives a reorg or silently credits money that no longer exists.
The tempting design is obvious: see the transaction in a block, store confirmations = 1, bump it on every new block, credit the balance when it crosses the threshold. It reads cleanly and it works — right until the chain reorganizes. Then your stored number describes a history that was discarded, and nothing in the data model knows that. The deposit keeps aging toward final on evidence that no longer exists.
In chain-watch, an open-source watcher I work on, confirmations are never stored as a running count. They are derived on every poll: current tip height minus the height of the block that includes the transaction, recomputed against the chain as it is right now. If that block is no longer on the canonical chain, the deposit falls back to unconfirmed on its own. There is no dedicated reorg handler, because there is no stale counter to repair. Reorg survival stops being a feature and becomes a property of how the state is computed.
The part people underestimate is what happens downstream. If a credited deposit later becomes orphaned, you cannot fix it by deleting the credit. In any ledger that has to be auditable, the original posting stays and you write a compensating entry against it, with a reference back to the transaction and the block that vanished. Otherwise you have a balance that changed with no explanation anyone can reconstruct six months later during a dispute.
So the rule I hold to: treat block inclusion as evidence, not as a fact. Evidence gets re-evaluated against the current tip every cycle. Facts get written once — and on-chain, almost nothing qualifies as a fact on first sight.
I write up the longer version of these design notes on my blog, including how the watcher models deposit state: https://polycratia.com/c/e9fcca71
If you run custodial deposits: where does your system store confirmation state — derived from the tip, or as a number you increment?
- $ · permalink
A coin selector that returns None is a bug report with the body deleted.
At 3am that null is the only thing standing between an operator and a stuck withdrawal, and it says nothing actionable. Underneath it are at least three different situations that need three different humans doing three different things.
Not enough total value: someone tops up the hot wallet.
Enough value, but it lives in dust: someone schedules a consolidation, ideally at a quieter fee level.
Enough spendable value, but the input that would cover the shortfall costs more in fee than it contributes: nobody should do anything except wait or change the policy.
Same null, opposite responses.
The second half of this is why selection cannot be a clean one-shot function at all. Fee depends on which inputs you pick. How many inputs you need depends on the fee. Estimate after selecting and you hand back a plan that stops covering itself the moment you add the input that pays for it. Fee estimation belongs inside the selection loop, and when the loop cannot converge, the loop is what knows why.
So: return the reason, not the absence.
How do you type selection failures in your wallet — enum, error subclasses, or a structured diagnostic the support tooling can read?
https://polycratia.com/c/ebeaffa6
- $ · 10 min readA balance is a query, and a hold is not a column
Custodial systems lose money in two ordinary writes: UPDATE accounts SET balance = balance + :amount, and the one next to it that nudges a frozen column up and down while a withdrawal is pending.…
- $ · permalink
A coin selection that returns None tells you nothing. Every support ticket after that starts with a guess.
I hit this building custodial BTC wallets. A withdrawal fails, and someone has to work out which of three unrelated situations they are actually in.
The user genuinely does not have the balance.
The balance exists but sits in dust that costs more to spend than it is worth.
The amount is fine, but the fee at the current rate exceeds the value being sent.
Same None. Three different responses: top up, consolidate, or wait and resize the transfer.
So in utxo-select I made the failure a first-class return value. Insufficient funds, dust-only inputs, and fee exceeding value are distinct outcomes carrying the numbers behind them, not a null. The caller decides what to surface to the user; the library refuses to flatten the distinction in the first place.
The broader lesson from years of payments work is that the empty result is where your operators live. Success paths get designed with care and failure paths get a null, and then the difference between user error and a fragmented wallet has to be reconstructed by hand during an incident.
The code is at github.com/polycratia/utxo-select if you want to see how the outcomes are modelled. I wrote up more of the reasoning behind decisions like this one here: https://polycratia.com/c/e36515ca
Where do you draw the line between a typed failure and a plain null in library code you maintain?
- $ · permalink
A balance column is the most expensive shortcut in a custodial system.
It reads like a fact. It is actually a cache — one you now have to invalidate correctly under concurrency, retries and partial settlement, forever.
I have built custodial wallets for BTC and ETH, a fiat-to-crypto onramp, and per-user virtual account ledgering where money arrived tagged only by requisites and had to be attributed before it could be trusted. The same pattern survived all of them: a balance is derived from paired postings, never stored. You append, you do not update. The number becomes a query, and it stays reconstructible from history instead of from a support ticket.
A hold follows from that. It is not a subtraction from an available column — it is its own object with a lifecycle: reserved, then settled or released, sometimes for less than it reserved. The moment partial settlement is possible, the two-phase object is the only model that stays honest.
And the money type should not belong to the ledger. Amount and currency as a protocol the ledger accepts is what lets card rails, account-to-account transfers and USDT postings land in one journal without the ledger knowing anything about any of them.
Where does your ledger keep the truth today — in a column, or in the postings?
https://polycratia.com/blog/a-balance-is-a-query-and-a-hold-is-not-a-column/
- $ · 9 min readDeclare once, apply to many: scoping VEX decisions without lying
A vulnerability scanner does not remember what you decided yesterday. Every build re-reports the same CVEs against the same dependencies, and someone re-reads the same advisory to reach the same…
- $ · permalink
The most dangerous field in a VEX document is the one filled in by a default.
When you generate VEX at scale, most findings are genuinely undecided. The scanner flags a CVE in a transitive dependency, and nobody has yet traced whether the vulnerable code path is reachable from your entry points. That is not a bug in the process. That is the normal state of a queue that is longer than the day.
The temptation is to make it disappear. Unknown reachability becomes not_affected, the document goes green, and the pipeline stops complaining.
But not_affected is an assertion. It says a human or a tool examined this and concluded the vulnerable function is never called. If nobody did that, the document is now carrying a judgement that does not exist anywhere in your organisation. The consumer downstream cannot tell the difference between analysis and a default value. That is how a compliance artifact turns into a liability.
I built vexdesk (github.com/polycratia/vexdesk) around the opposite rule: an undecided finding is emitted as under_investigation, never quietly resolved. The status is uncomfortable to look at, and that is the point. It puts real work on a real queue instead of hiding it behind a green field.
I have made the same trade before in payment reconciliation. When an incoming payment cannot be matched to an obligation with confidence, you do not attach it to the closest plausible account. You flag it as unmatched and let a human decide. The unmatched pile is ugly, it is visible, and it is honest. Every system I have seen that auto-guessed instead ended up with balances that were silently wrong for months.
The general principle: a system should never manufacture certainty it was not given. "I do not know yet" is a legitimate output, and an artifact that cannot express it will lie by omission.
For anyone generating VEX or SBOM attestations in CI — how do you keep the under_investigation pile from becoming a permanent parking lot?
- $ · permalink
Minting a token that represents a private aircraft is the easy part. Deciding what makes that token wrong is the hard part.
I built NFT generation and lifecycle on Ethereum for high-value physical assets, where the token accumulates signed documents and deal state from listing through closing. The instinct is to treat the token as the source of truth. It isn't. The asset exists in the physical world, the obligations exist in signed paper, and the chain only knows what someone chose to write to it.
So the token drifts. A deal stage advances off-chain, a signature is collected but never anchored, a party walks away — and on-chain state now claims something the world no longer supports. Nothing reverts it, because the chain is append-only and perfectly happy holding a stale claim forever.
What worked: no state transition without a signed document behind it. The document and signature live off-chain in the workflow system; the chain holds the commitment and the ordering. The token becomes a log of attested events, not an assertion of fact.
Most of the engineering in an RWA deal turns out to be e-signature and document lifecycle. The chain is the smaller half.
If you're building RWA flows: what do you anchor on-chain, and what stays in the document system?
- $ · permalink
A balance is not a number you edit. It's a number you derive.
The moment a ledger stores balance as a column and updates it in place, every reservation becomes an UPDATE — a pending payout, an investor commitment waiting on moderation, an on-ramp order not yet confirmed. You get a correct-looking number and no way to answer why it is that number.
So I treat a hold as its own journal object. Written once, with an amount, an owner, and a state. Later it settles into a real posting, or it releases. Nothing in between rewrites history, and available balance becomes derived: posted entries minus active holds.
The case that breaks naive implementations is partial settlement. A hold rarely settles for exactly what was reserved — an authorization captures less, a payout goes out net of fees. That is two facts, not one edit: post the actual amount, release the remainder, both traceable to the same hold.
The honest trade-off is read cost. Deriving a balance from full history is slow, so you need checkpoints — and a checkpoint is a cache, with all the invalidation problems caches have.
I pulled the pattern into a small reference implementation at github.com/polycratia/ledger-core, mostly because I kept re-explaining the same design in reviews.
If you run holds in production: what closes them — the provider callback, or a sweeper you trust more than the callback?
- $ · 7 min readMulti-carrier shipping rates are quotes, not prices
The shipping amount a customer sees at checkout comes from a carrier API call that happened seconds or minutes earlier, computed from package dimensions and an address that can both change before…
- $ · permalink
Everyone building a custodial wallet worries about key storage. The part that actually breaks is the withdrawal queue.
Key management is a solved problem with known answers. Withdrawals are not, because they span two systems that cannot share a transaction. Your ledger debit is reversible. The broadcast is not. Between those two moments sits every incident I have seen in custodial systems since 2018.
A node call times out and you do not know whether the transaction was broadcast. A retry looks safe on the application side and is catastrophic on chain. A user's balance drops before finality and rises again after a reorg. A nonce is reserved by one worker and consumed by another.
The approach that has held up for me: the ledger debit and the intent to send are written in the same transaction, and nothing broadcasts from that path. A separate worker owns broadcast, owns the nonce, and is the only thing allowed to move a withdrawal forward. The chain state is treated as an external fact that gets observed and recorded, never as something the request path can assume. Every state change is append-only, so a stuck withdrawal can be read as a history rather than guessed at.
The useful reframe: a withdrawal is not an action. It is a long-running agreement between your ledger and a chain that does not know your ledger exists.
For those running custody in production — where does your withdrawal path lose certainty first: broadcast ambiguity, nonce contention, or confirmation depth?
- $ · permalink
KYC is not a checkbox at signup. It's a long-lived state machine you have to keep running.
Most systems I've picked up model it as a boolean on the user row: verified, true or false. That holds until the first real case.
A provider returns "pending manual review" while the user is already mid-deposit. A document expires while the account stays open. A re-check months later comes back with a different risk decision. A compliance officer overturns an automated pass.
Each of those is a transition. If your schema stores only the last answer, you lose the reason and the ordering — the exact two things you get asked about later.
When I built a fiat-to-crypto onramp with real-time moderation, the hard part was never the provider integration. It was this: every decision written as an immutable event with its source recorded (provider, internal rule, or human), the user's status derived from that history rather than overwritten in place, and permissions gated per action instead of globally. Deposit, withdraw, and raising a limit are not the same bar.
The payoff shows up on the day a case is disputed. You replay the history instead of reconstructing it from application logs.
If you run verification in production: does your system store the decisions, or only the current verdict?
- $ · 10 min readCustodial ETH withdrawals: the nonce is a database row, not a node call
A custodial wallet service has to turn a user's withdrawal request into exactly one on-chain transaction, while every layer around it — HTTP clients, queues, restarts, impatient users — is…
- $ · permalink
The blockchain part of a custodial wallet is the easy part. The hard part is deciding when a deposit becomes money the user can spend.
Watching addresses and parsing transfers is a solved problem. What is not solved for you is finality. A transaction you saw in the mempool can vanish. A block you credited against can be reorganized away. And the user is already withdrawing.
When I built custodial BTC and ETH wallets and an on-chain payment system, the design that held up was to stop treating "deposit detected" and "balance available" as the same event. Detection creates a pending ledger entry. Confirmation depth promotes it. Only the promotion touches spendable balance, and the promotion is a normal, replayable ledger operation rather than a special case.
That gives you one thing that matters more than speed: a reorg becomes a compensating entry, not an incident. You already have the vocabulary to reverse it, and the history shows what happened instead of a silently corrected number.
The trade-off is user-visible latency, and every product wants to shorten it.
If you run custodial balances: where does your confirmation threshold come from — the asset, the deposit size, or the user's withdrawal behaviour?
- $ · 8 min readThe only safe failure state for an outbound payout is unknown
Sending money out of a system is not the mirror image of taking it in. An inbound payment can be re-parsed, replayed and reconciled at leisure; an outbound payout that leaves twice is a loss…
- $ · 8 min readSplitting a loan repayment across investors without losing a cent
A single borrower repayment often funds a loan held by many investors at once, and the split has to be exact: every minor unit that arrives has to leave the account assigned to somebody. Percentage…
- $ · permalink
Signing a transaction is the easy part of a custodial wallet. Deciding when the money actually left is where the system breaks.
I have built custodial BTC and ETH wallets and on-chain payment flows, and the recurring failure is not cryptography. It is that you now run two ledgers with different notions of finality. Yours is transactional and instant. The chain's is probabilistic, reorganisable, and occasionally silent for hours.
If a user balance is derived from chain state, every confirmation-depth choice becomes a product decision made by an engineer at 2am. If it is derived from your own ledger, you have to explain every gap between what you recorded and what the chain settled.
What I do now: the internal ledger owns the balance, and the chain is treated as an external settlement source that gets reconciled against it — same way I would treat a bank's statement in a fiat payment system. A withdrawal moves through intent, broadcast, and confirmed as separate recorded states, never one boolean. Stuck and replaced transactions then become normal cases with a place to live, not incidents.
The useful reframing: on-chain is not your database. It is a counterparty you reconcile with.
For those running custodial flows — do you let confirmation depth vary by amount, or keep one fixed rule and eat the tail risk?
- $ · permalink
Detecting an incoming crypto deposit is the easy part. Deciding when it is real is what breaks custodial systems.
Your node tells you a transaction exists, then that it landed in a block. Neither fact means a user balance should move. How many confirmations you wait for is a risk decision — how much you are willing to lose against how long a user will tolerate a spinner — but in most systems it ends up as a constant hardcoded by whoever wrote the deposit watcher first, identical for every asset and every amount.
Then a reorg happens, and you find out your ledger has no state for "credited, but still reversible". If the only states are pending and settled, un-crediting becomes a manual adjustment. And if the user already withdrew, the adjustment is just a record of your loss.
How I build it now: the watcher records observations, never balances. A separate crediting policy decides — per asset, per amount — when an observation becomes a ledger entry. Reversals are ordinary ledger entries, not admin surgery.
The chain is the source of truth about transactions. It is never the source of truth about your balances.
Where do you draw the confirmation line — fixed per asset, or scaled with the amount at risk?
- $ · 8 min readTreat LLM Catalog Translation as a Cache, Not a Job
Translating a product catalog with an LLM looks like a batch job: read rows, call the model, write rows back. It stops looking like that the first time a supplier re-uploads the same feed and you…
- $ · 7 min readKYC webhooks arrive out of order, so rank the decisions
An identity verification provider does not hand you a status. It hands you a stream of webhook events, and that stream arrives out of order, duplicated, and occasionally after one of your own…
- $ · 8 min readCrediting On-Chain Deposits Without a Balance Column
A custodial wallet has to answer one question honestly: how much of this user's money is spendable right now. The obvious implementation increments a balance column when a transaction appears in a…
- $ · permalink
A catalog parser that crashes is a good day. The expensive one keeps returning valid data that no longer means what it used to.
I have run retail catalog parsers for a cross-border shopping platform for years, and the pattern repeats. The selector still matches. The price is still a number in range. But the page started showing a member price instead of the retail one, or the unit shifted from a pack to a single item, or a variant field moved and every size now maps to the default. Nothing throws. Downstream, the listing is wrong, the delivery rate is quoted against the wrong weight, and you find out from a customer rather than from a log.
So I stopped treating parsers as code that either works or errors, and started treating extraction as data with an expected shape. Field-level presence rates, value distributions per source, and a diff against the previous crawl. A sudden jump in the share of products with no discount, or a currency that appears where it never did before, is a stronger signal than any exception.
Structural change is easy to detect. Semantic change is the one that costs money.
If you run scrapers at scale: what actually tells you a source changed meaning rather than markup?
- $ · permalink
Most teams build KYC as a gate. The user passes, you flip a boolean, and you move on.
That boolean is the bug.
Verification is not a fact about a person. It is a claim made by a provider, at a specific moment, based on documents that were valid then. Documents expire. Sanctions lists change. Risk scores get revised. A user who cleared verification in March may not be clearable today, and nothing in your system will tell you that, because you stored an answer instead of a decision.
When I build compliance tooling now, I store the whole decision: which provider responded, what they returned, which ruleset version was applied, and when. Approval becomes a dated record, not a permanent property. Re-checks are scheduled, not triggered by someone noticing.
The practical payoff shows up on the moderation side. Your compliance team stops asking "is this user verified?" and starts asking "what did we know, and when did we know it?" — which is the only question that survives an audit.
The part that always causes an argument: what happens to an in-flight transaction when a user's status goes stale mid-flow. Block it, let it settle, or hold and escalate?
How do you handle that one?
- $ · 9 min readMatching Incoming Bank Transfers to Users When the Reference Is Wrong
A bank statement line gives you an amount, a date, a sender, and a free-text reference field, and your system has to decide which user that money belongs to. The reference field is wrong, truncated…
- $ · permalink
The hardest part of a custodial wallet is not key management. It is deciding the moment a deposit becomes real.
Key handling is a solved problem: isolate the signer, keep it off the API path, limit who can trigger it. Tedious, but well understood.
The ambiguity lives on the credit side. A transaction in the mempool is not money. One confirmation on one chain does not mean what it means on another. A reorg can take a deposit back after you have already shown the user a bigger number. Product wants instant credit; the ledger wants finality. Those two pull in opposite directions, and the gap between them is where custodial systems quietly lose money.
What I do: keep pending and confirmed as separate ledger states, and let only confirmed balance be spendable. Every credit is idempotent on the chain identifiers of that specific output, so a node rescan or a replayed block cannot double-credit anyone. Confirmation depth is a policy value per asset, not a constant buried in code. And a deposit that reorgs out gets a compensating entry, never a deletion.
The chain is an input. Your ledger stays the source of truth.
Where do you draw the instant-credit line — a fixed confirmation depth, or tiered by amount?
- $ · permalink
Translating a product catalog with an LLM is not a translation problem. It is a cache invalidation problem.
The model call is the cheap part. The hard part starts the moment the source catalog moves, and retail catalogs move constantly. A title gets a new size suffix, a supplier rewrites a description, an attribute appears that was never there before. Now you have to decide which of your translated fields are stale and which are still fine.
Re-translate everything on every sync and your cost and latency scale with the size of the catalog instead of the size of the change. Re-translate nothing and the storefront quietly drifts away from the source until someone finds it in a support ticket.
What works for me: store each translation keyed on a hash of the exact source fields that fed the prompt, plus the prompt version and the target locale. Change any of those and the entry is invalid. Change none of them and you serve what you already have. The model becomes a fill-on-miss function behind a cache rather than a stage in the pipeline.
The useful side effect is that prompt changes become reviewable. A new prompt version is just a new key, so you can roll it through a slice of the catalog instead of rewriting the whole store in one job.
If you run LLM pipelines over data that keeps changing: what do you key on, and where does that key first betray you?
- $ · permalink
A crypto deposit is not an event. It is an opinion that gets stronger over time.
Most custodial wallet code treats it like an event anyway: watch the chain, see the transaction, credit the balance. That works until the block it landed in stops existing.
What I learned building custodial BTC and ETH wallets is that the interesting design decision is not the confirmation threshold. It is what the user can do with money that is only probably theirs.
So I model a deposit as a state, not a notification. Detected, pending, credited, reversed. Each state has its own rules about what it unlocks: visible in the interface early, spendable late, withdrawable off-platform latest of all. A reorg then becomes a normal transition instead of a support ticket and a manual database edit at midnight.
The part teams skip is the reversal path. Everyone writes the credit logic. Almost nobody writes the code that takes it back and leaves an auditable trail, because it feels like an edge case until the day it is not.
If you run custodial balances: do you let users trade on unconfirmed deposits, and if so, who eats the loss when the chain disagrees?
- $ · permalink
KYC is not a signup step. It is a state that changes over the whole life of the account, and most systems store it as a boolean set once.
That boolean is where things break later. A document expires. A provider re-runs screening and the result flips. A compliance officer reverses an automated approval after manual review. Sanctions and watchlist data changes underneath users who were cleared months ago. Nothing in the signup flow ever notices, because the flag was written once and never asked again.
Building onramps and verification tooling, I stopped storing a verification result and started storing a history: each decision as its own record, with the source (provider or human moderator), the reason, and the period it is valid for. Current status is derived, never overwritten. Manual moderation writes a new decision instead of editing the old one.
Then every money-moving action asks the same question at execution time — is this user cleared right now — instead of trusting a flag set at registration.
The practical payoff is not elegance. It is that when a regulator or a partner asks why a specific withdrawal was allowed on a specific day, you can answer with data instead of a guess.
For those running verification in production: what finally forced you to make KYC status time-aware — an expiry, a reversal, or an audit?
- $ · permalink
Integrating a shipping carrier is not an API mapping problem. It's a state problem you don't control.
Every carrier exposes its own status vocabulary, and the naive move is to map each one onto your own enum and call it done. That works until you run several carriers at once. Then you learn that events arrive out of order, that a shipment can move backwards from out-for-delivery to in-transit, that some carriers repeat a status for days and others go silent for a week, and that delivered is not terminal because a return can reopen the whole thing weeks later.
What I ended up doing on cross-border delivery pipelines: store the raw carrier event exactly as received, with its own timestamp, and treat it as an append-only log. Derive my status from that log rather than overwriting a status column. Order by the carrier's event time, not arrival time. And keep an explicit unknown state instead of forcing every foreign status into a bucket that looks tidy but lies to support and to the customer.
The rule I'd keep: never let an external system write directly into your state field.
For those running multi-carrier shipping — what broke first for you: status mapping, silent gaps, or returns after delivery?
- $ · 6 min readTranslating a product catalog with an LLM: cache keys and guard rails
The first version of an LLM catalog translation pipeline usually works and is still wrong. It re-translates fields nobody touched, and it happily publishes a description where the model rewrote a…
- $ · permalink
The hardest part of a custodial wallet is not key management. It's that a withdrawal is a distributed transaction between your database and a chain that will never roll back for you.
The naive version looks fine in review: open a transaction, debit the user's balance, call the node to broadcast, commit. It survives testing because nothing fails in testing.
In production the node call times out after the transaction is already in the mempool. Now you either roll back and pay out twice, or commit and hope. Neither is an answer.
How I build it instead: the debit and a pending withdrawal record commit together, and nothing touches the chain inside that transaction. A separate worker picks up the pending row, broadcasts, and moves it through a state machine — pending, broadcast, confirmed, failed — keyed by the transaction hash. Broadcasting the same signed payload twice is harmless. Signing a second one is not, so the nonce or UTXO selection has to be owned by one writer.
The reconciler then walks confirmed chain state back against the ledger, and anything that does not match gets flagged rather than auto-corrected.
The rule I keep coming back to: never let an irreversible external effect share a commit boundary with your own state.
For anyone running custodial withdrawals — where does yours get stuck most, stuck broadcasts or the fee bumping?
- $ · permalink
In payments, the retry is not the edge case. It's the normal case.
Every hop between you, the bank, and the card network can time out after the money already moved. The request succeeded; the response got lost. So the client retries, the webhook fires twice, the reconciliation job replays a batch — and if each of those can charge or credit again, you will double-process. Not might. Will.
The mistake I see most is treating idempotency as an API-gateway concern: dedupe the incoming request by a header key and call it done. That catches the naive double-click and nothing else. The dangerous duplicates come from inside your own system — a queue redelivering, a cron rerunning after a crash, two workers grabbing the same row.
So I push idempotency down to where the money actually changes: the ledger write. Every financial effect carries a deterministic key derived from the business event, not the transport. Applying the same key twice is a no-op that returns the original result. The API layer, the queue, and the retrying client can all be as sloppy as reality forces them to be, because the last line of defense is the one place that can't afford to be wrong.
The trade-off is real: those keys have to be stable across code changes, and picking what goes into them is a design decision you live with for years.
Where does duplicate processing bite you first — provider webhooks, your own message queue, or replayed batch jobs?
- $ · permalink
A Python 2 to 3 migration is not a syntax problem. It's a data problem wearing a syntax costume.
The automated tools rewrite prints and imports in an afternoon. What they cannot touch is every place the old code treated bytes and text as interchangeable: file reads, socket payloads, database drivers, anything pickled or cached back when nobody had to decide.
Under 2 that ambiguity got resolved at runtime, usually by accident. Under 3 it becomes a boundary you have to declare explicitly, and the codebase never recorded where those boundaries were.
The failures don't surface in tests. They surface on the one row with an unusual encoding, the legacy record written by a service that no longer exists, the comparison that used to work because 2 would order anything against anything.
How I approach it: map the I/O edges first and pin encoding at each one, then run both versions against production-shaped data and diff the outputs instead of trusting a green suite. Migrate boundary by boundary, not module by module.
Nothing in the language will point at those edges for you. Only real data will.
If you've been through one: where did yours break first — the ORM layer, the caches, or file and CSV handling?
- $ · permalink
The first time a payment integration bit me in production, it wasn't a failed charge. It was a successful one that ran twice.
Every link in a payment chain retries: the client on a dropped connection, the queue on a timeout, the provider's webhook that fires again because your 200 got lost on the way back. None of them know the operation already happened. At-least-once delivery is the default you inherit whether you designed for it or not.
So the job was never "make sure this runs." It was "make sure running it three times leaves the same result as running it once."
The way I approach it now: an idempotency key minted at the true origin of the intent, not regenerated per retry. A persisted record of that key and its outcome before any side effect touches money. Second attempt looks up the key, sees the settled result, and returns it instead of moving funds again. The dedup has to live in the same transactional boundary as the write it guards, or you've just moved the race one layer down.
The trap is scoping the key too narrowly and treating a legitimate second payment as a duplicate, or too broadly and swallowing one the user actually meant to send twice.
Where does your idempotency boundary sit — at the API edge, the queue consumer, or the ledger write?
- $ · permalink
A catalog scraper almost never fails loudly. It fails silently, and that's what makes it dangerous.
When you pull product data from retail sources you build for the happy path: fetch the page, match the selectors, extract price, stock, title, images. It works, so you move on. Then the source ships a redesign, or splits one field into two, or starts A/B testing a new layout on half its traffic. Your scraper keeps returning 200. It keeps writing rows. Nothing throws. But the price is now the crossed-out price, or stock reads "in stock" from a cached banner, and your catalog is quietly wrong for days before a human notices a listing that makes no sense.
After years running parsers across many retail sources, I stopped treating extraction as the hard part. Fetching is easy. The real work is trusting what you fetched.
So I build the parser to distrust itself: shape checks on every extraction, ranges each field must fall in, and a diff against the last known-good snapshot per source. A sudden jump in null rates or a price that moved an order of magnitude doesn't get written — it gets quarantined and flagged. I'd rather serve stale data I know is stale than fresh data that's silently corrupt.
For those running scrapers at scale: what's your first line of defense against a source that changes shape without telling you?
- $ · permalink
Minting an NFT for a physical asset is a weekend job. Making it survive a real deal is the part nobody warns you about.
I built NFT lifecycles on Ethereum for high-value physical assets — private aircraft, where a single transaction runs from listing to closing over weeks. The token isn't a picture. It's a container that has to accumulate signed documents, escrow state, compliance checks, and multi-party approvals, all while staying consistent with an off-chain reality that keeps moving.
The hard questions are never on-chain. What is canonical when a document is signed off-chain but the token hasn't caught up? Who can advance the deal state, and what stops a stale client from advancing it twice? Where does the escrow release actually gate?
My rule: the chain records agreement, not truth. Truth lives in a reconciled backend that treats every on-chain event as a claim to verify, not a fact to trust. The token references state; it doesn't own it.
If you've put real assets on-chain: where did your on-chain and off-chain state first drift apart, and how did you catch it?