build log
- $ · 12 min readA JWKS cache keyed on the token, not on a timer
A key set cached for fifteen minutes stays correct right up to the moment the issuer rotates. An issuer signs with a new key as soon as it publishes it, so a resource server waiting out its TTL…
- $ · permalink
Your database is fast and your API is slow. Both can be true at once, because most of a request's life happens outside the database.
A query timer starts when the statement reaches the engine and stops when the last row comes back. Everything around it is invisible to that number: waiting for a free connection in the pool, turning rows into objects, serializing them, compressing, handing bytes to a proxy, then pushing those bytes across whatever network the client is actually sitting on.
The wire is where API latency hides most often. A response the server finishes in milliseconds can land seconds later at the client if it carries a deeply nested payload over a mobile link. The server measures time to first byte. The user lives with time to last byte. No database dashboard knows the difference between the two.
In payment integrations this shows up as an argument nobody can win: the merchant reports a slow endpoint, my own logs show a fast one, and both parties are telling the truth. The query was never the subject.
So I budget latency per response rather than per query. Response size in bytes, per endpoint. Serialization time instrumented separately from data access. Time to last byte measured at the edge, not in the handler. Once those three numbers exist, trimming fields and paginating stop being cosmetic cleanups and become performance decisions with a known payoff.
The other gaps in that chain (pool checkout, proxy buffering, serialization cost) I write up in more depth on my blog: https://polycratia.com/c/b5cfa43b
If you've chased this before, you probably already know which gap turned out to be yours: the pool, the serializer, or the bytes...
- $ · permalink
Three seconds in an API is rarely three seconds of work. Usually it's one request waiting on a lock that another request holds across a network call.
The shape shows up all over payment code: open a transaction, take an exclusive lock on the balance row, call the provider, write the result, commit. Correct, readable, and it holds that row for the whole provider round trip.
At one request per second nobody notices. Put twenty requests on the same wallet or the same merchant account and latency goes serial: each one waits for every provider call ahead of it. The database is idle. CPU is idle. The slow query log is empty, because nothing was slow. Things were queued.
The tell: latency tracks concurrency on a single key, not data volume. The endpoint is fine for most of your users and awful for your busiest one, with identical SQL.
How I handle it now: no I/O I don't control inside a transaction. I record the intent and commit. The provider gets called outside any lock. Then a short transaction, keyed by an idempotency key, to apply the outcome. Where the ledger allows it I append entries instead of mutating one row, so writers stop competing for that lock at all.
Lock hold time is a latency budget you spend on behalf of every concurrent caller. I keep two explicit lists, operations that may hold a lock and operations that may touch the network, and I don't let them overlap.
I wrote up more of these failure modes on the blog: https://polycratia.com/c/a97c2017
Worth knowing where your hottest row lives: a user balance, an order, or a counter...
- $ · permalink
The most expensive bugs I have seen were not in business logic. They sat in a comparison.
Logic errors announce themselves: something throws, a test goes red, a flow stops. A bad equality check does none of that. Both sides are valid strings. The comparison runs, returns false, and the system carries on exactly as designed.
Identifier matching is where this lives. An Ethereum address stored in mixed-case checksum form in one place and lowercase in another. A payment reference carrying a trailing space from a bank file. An order id with a leading zero that someone cast to an integer on the way in. One character of difference, zero errors raised.
The failed match itself does little damage. The damage is what that failed match looks like downstream: a deposit that is on-chain, confirmed, irreversible, and simply not attributed to a user. Support hears about it before monitoring does, because your success rate stays flat. You handled the event. You handled it into the wrong bucket.
A more careful code review does not fix this. What fixes it is removing the opportunity to compare raw input at all: normalize every identifier at the system boundary, persist the canonical form, and enforce it with a database constraint so a non-canonical value cannot be written in the first place. Comparison then has nothing left to get wrong.
Treat any identifier crossing a system edge as untrusted format, not just untrusted data.
I go deeper into failures like this on the blog: https://polycratia.com/c/26e97fd3
I am curious where format drift enters your system first: provider webhooks, bank statement files, or your own admin panel...
- $ · permalink
Nobody had edited the product, and it kept showing up as updated several times a day. The pipeline reported success every single time.
I was running an LLM pipeline that translated and adapted a retail catalog for another market. In testing it looked clean: feed in a product, get back a title and description, compare against the expected output, pass.
Production added one thing my fixtures never had: a second run over the same unchanged product.
The model is not a pure function. Same input, slightly different wording. Not wrong. Just different.
And everything downstream read "different" as "changed". The search index reindexed. Change feeds fired. The notification service fanned out to people who had subscribed to that product. Caches invalidated for no reason.
Nothing threw an exception, every stage was green. The output itself was not broken. What it cost me was write amplification through half the system, triggered by a pipeline that was technically working.
What I do differently now: the agent's output is an artifact with an explicit key, built from normalized source fields, prompt version, model version. If the key hasn't changed, I don't call the model at all, I serve the stored result. If the key has changed but the new text is semantically equivalent to the stored one, downstream still sees no change.
The test I was missing had nothing to do with output quality. It was this: run the pipeline twice over identical input, then assert the second run produces zero writes.
An agent you evaluate one call at a time tends to look fine. The failure lives in the second call.
I write more about structuring these pipelines for production here: https://polycratia.com/c/19065af8
So where did your agent first surprise you: in its output, or in what its output triggered downstream?
- $ · permalink
Compliance staff waited about three seconds every time they opened the moderation queue. The rows on screen took tens of milliseconds.
It was a KYC review queue: applications, filters on status and date, fifty per page. I profiled it expecting a bad join or a missing index. The page query was fine. Almost all of the wait came from one line that had gone unexamined, the total count behind the pager.
Each filter combination made that count walk the whole filtered set, just so the interface could print "page 1 of 1,400". Showing fifty applications was cheap. Knowing how many applications existed was the entire response time.
So I stopped answering that question. Keyset pagination on created_at plus id: the next page is a seek from the last row you saw, not an offset the database has to count past. The total became a capped probe. Look for a bit more than a thousand, and if they are there, the interface says 1000+. Three seconds down to roughly 200ms, and deep pages got faster instead of worse, which offset pagination does not manage.
The code took an afternoon. The conversation took longer. Someone has to accept losing jump-to-page and the exact total. In an internal work queue that was easy: moderators rarely jump to page 47, they filter. In a customer-facing catalog I would have argued much harder, or kept an approximate count maintained on write.
What I take from it: a lot of the latency I remove is not work done badly. It is work few people need, rendered in a control that mostly sits unused.
I write these up as I hit them, with more of the reasoning than fits here: https://polycratia.com/c/6f8aea57
If your list endpoints return an exact total, worth checking whether that is a real product requirement, or whether it probably arrived with a UI component...
- $ · permalink
Before a rewrite, the questions worth asking are about the overlap period, not about the new architecture.
The target design is the cheap part. You can draw it in an afternoon, and most senior engineers will draw something defensible. Rewrites die in the months where both systems are alive and the data has two authors.
So the five questions I actually ask:
-
During the overlap, which system authors each table? If both write, the invariant has to be enforced in exactly one place. Otherwise two correct state machines produce one incorrect ledger.
-
Can you compare outputs instead of trusting tests? Run the new path in shadow against real traffic and diff the results, including the inputs nobody designed for. Tests encode what you already understood.
-
Can you move one flow instead of the whole service? If the smallest shippable unit is "all of it", the seam is wrong. Pick a seam where one caller and one write path line up with one owner.
-
Can you reverse a flow after it has written rows? That means retained raw inputs and version-stamped derived data. Without those, rollback is a rewrite of its own.
-
What proves the old path is dead? Something has to count calls per caller. Otherwise an admin panel or a nightly job keeps writing through the code you believe you retired.
None of this is about code quality. It is about who authors data and how you observe divergence. Also whether you can undo a step.
I have run Python 2 to 3 migrations and monolith-to-services rewrites on systems handling money, and in every one the hard decisions were migration-order decisions rather than design decisions.
I wrote up how I sequence this, shadow-and-diff setup included, here: https://polycratia.com/c/c826fffe
If you have done a rewrite on a live system, tell me what finally told you the old path was safe to delete...
-
- $ · permalink
An agent that passes its tests has only ever seen two outcomes from a tool call: success and failure. Production has a third one: unknown.
The call times out. The charge may or may not exist. The transfer may or may not have been submitted. Nothing in the agent's context separates "did not happen" from "happened, and the answer got lost on the way back". In an eval harness that gap never opens, because the stub always answers.
So the agent does the reasonable thing and retries. In a read-only world a retry is free. On money rails a retry is a second intent.
The repair is not a better prompt. It is tool design.
Every writing tool takes an intent key supplied by the caller, and a replay returns the original outcome instead of performing a new one.
Every tool declares whether it is reversible. Irreversible calls do not get blind retries: they get a reconcile step that asks the provider what happened to that intent, and only then decides.
"Unknown" becomes a value the agent can hold and act on, not an exception string it has to interpret.
What that buys you is simple: the retry turns into a read. Correctness stops depending on the model's judgement about an ambiguous error, and that is precisely the part you cannot test, because it varies run to run.
Most reliability work in production agents does not sit in the model. It sits in making the surface it acts on replay-safe.
I went deeper on how I design that layer here: https://polycratia.com/c/01a7696b
If you are running agents against real side effects, it is probably worth checking whether your tool layer returns "unknown" or just collapses it into an error.
- $ · permalink
A signed document came back invalid, and the logs said nothing about why.
Multi-party signing flow on a high-ticket asset deal: each signature attaches to the deal record and has to stand up later. On my machine it signed and verified clean, run after run. In production it failed occasionally, and not twice in the same place.
The signature carried a validity window. The host that minted the timestamp and the host that verified it were not the same host, and their clocks were not the same clock. Some signatures came out a moment in the future relative to the verifier, so the verifier refused them, correctly: not yet valid.
My laptop had one clock, so that whole class of bug was invisible there. Containers don't help you either, because a container still reads the host clock.
The cost wasn't compute. It was documents parked mid-workflow, and counterparties asked to sign again for no reason they could see. In a deal flow that reads as a credibility problem, not a bug report.
What I do differently now: time in a signing or expiry path is data with a source, not something a process invents. One authority issues it, usually the database, and each window carries an explicit tolerance, written down instead of inherited from whatever drift happens to be true that day.
The rule I took from it is broader than clocks. Anything my machine has exactly one of (one clock, one process, one worker, one region) is a guarantee I'm depending on without having chosen it.
I write these up in longer form here: https://polycratia.com/c/e931c899
Curious which single-instance assumption bit you hardest: clock, process, worker count...
- $ · permalink
A logging call threw inside a payment webhook handler. By morning I was reconciling a day of payments by hand.
The handler did the real work first, and in the right order: verify the signature, persist the event, credit the virtual account. Then it fanned out a notification and wrote an audit line, and that last part read a field that happened to be null for one payment type.
The exception escaped the handler. The framework turned it into a 500. The provider did what its documentation says it does: retried, got another 500, and after a streak of failures it stopped delivering to that endpoint at all.
Nothing in my monitoring fired. A handful of 500s sat well inside normal noise. The database was healthy. The queues were empty, because nothing was arriving. Incoming payments just stopped being attributed, and balances froze at a number that had been correct the previous evening.
Two things changed after that.
The handler now has one job: verify, persist the raw event, return 200. Anything derived from that event runs as its own retried step, where a failure is mine to fix and not a reason for a provider to disconnect me.
And I monitor absence. Time since last event, per provider, per channel, with a threshold that pages. A dependency that has quietly stopped talking to you emits no errors, and every alert I owned was watching for bad things happening. None of them watched for nothing happening.
I wrote more about the attribution and reconciliation side of this on my blog: https://polycratia.com/c/f91de547
You probably have one integration that could go silent for a full night before anyone noticed...
- $ · permalink
A 3-second response is usually not slow work. It is a failure that succeeded on the third try.
Latency that lands on a round number is rarely computation. Computation is messy, and it scales with how much data you throw at it. Round numbers come out of configuration: a backoff curve, a retry count, a client default nobody picked.
The mechanics are simple and invisible. Your handler calls a provider (a card gateway, a verification vendor, a chain node). The first attempt dies on a connection reset or a 502. The HTTP client, following defaults it inherited, waits half a second and retries, waits a second and retries. The third attempt works. The handler returns 200.
Nothing lands in the logs as an error, because nothing finally failed. So the error rate looks clean and the endpoint looks slow. You go profile the database, and the database is innocent.
What fixes this is not a faster retry: it is making the retry budget visible.
One deadline per inbound request, passed down to every outbound call, so the attempts can't add up to more than the caller will wait.
Retries only where repetition is safe: idempotency-keyed, not "probably fine".
Attempt count as a metric on every provider call, so a rising retry rate shows up before it turns into latency.
In payment and verification flows, a success that took three seconds carries more information than a clean failure. The failure tells you something broke. The slow success tells you something is breaking, repeatedly, and your instrumentation agreed to hide it.
I write up more of these production patterns on my blog: https://polycratia.com/c/e6ef3984
Worth knowing where retries live in your system today: a client-library detail, or a number you can alert on...
- $ · 8 min readStatus 1 proves the transaction ran, not that the tokens moved
A mined ERC-20 transfer hands you two witnesses, and they do not say the same thing. The receipt status tells you whether execution reverted. The Transfer log is the chain's own record that a…
- $ · permalink
Your agent did not get worse in production. It just stopped being the only thing changing the world.
In testing it plans against a frozen snapshot: one caller, one request at a time, with state that holds still between the decision and the action. Every eval you run rewards exactly that.
Production takes the freeze away. Between the moment the agent reads state and the moment it acts, a human in an admin panel edits the record, or a provider callback lands, or a retry from your own queue shows up out of order. The reasoning is still correct. The world it referred to has expired.
So this is not a model quality problem, and more evals will not surface it, since evals replay a stable world too.
I stopped treating agent output as a command and started treating it as a proposal. The proposal carries the state it was computed on: record versions, the balance it read, the status it assumed. Whatever service applies the effect revalidates those preconditions inside the same transaction that commits it, and rejects the proposal if anything moved. A rejected proposal is cheap, you re-plan. A proposal committed against stale state is a wrong write you now have to go find and unwind.
The other half is the tool boundary. Every side-effecting call needs an idempotency key derived from the plan step and not from the retry attempt, or your retry logic quietly turns into a duplicate-payment generator.
Agents inherited the hard parts of distributed systems, and most of them, I think, are still tested like pure functions.
I unpack how I structure the proposal-and-revalidate layer here: https://polycratia.com/c/33165120
If your agent touches money or inventory, the thing to pin down is where you revalidate state: at plan time, or at commit time...
- $ · permalink
Three seconds is almost never a performance number. It's a constant somebody configured.
Work-bound latency is messy. It spreads out, it drifts with load, it has a long tail. So when response times cluster tightly on 1s, 3s, 5s, you are not watching a system compute: you are reading a timeout back out of your own config, usually multiplied by a retry count.
Anything that fans out to third parties shows this best. A delivery-rate lookup that asks several carriers for quotes answers no faster than the slowest one you agreed to wait for. One unhealthy endpoint doesn't make the response a bit slower, it pins it to the ceiling, every time, for every user. An HTTP client with default retries turns a one-second ceiling into three. And since connect timeout and read timeout are separate knobs, a dead host and a thinking host fail on different clocks, which is why your graph shows two plateaus instead of a curve.
So the useful question isn't what's slow. It's what's waiting, and on whose schedule.
How I handle it now: the request gets a total budget, each dependency gets an explicit deadline carved out of what remains, and partial results count as a real response. Rates from four carriers out of five, flagged incomplete, beats a complete answer nobody stayed for. Then I read histogram shape instead of averages. Plateaus are configuration. Slopes are work.
I write up more of these production patterns on my blog: https://polycratia.com/c/e0fb1559
And when your p95 lands on a suspiciously round number, probably worth knowing which dependency owns that clock...
- $ · permalink
The slowest endpoint in that service did the least work: a small config read that touched no database at all.
Everything I looked at said the database was healthy. Queries in single-digit milliseconds, no lock waits, connection pool half empty. So I added workers, the easy move. Latency got better for about an hour, then came back exactly where it started.
The service was async. One path through it was not.
A provider SDK sitting in the payment flow made its HTTP calls synchronously, and that call lived inside an async handler. While it waited on the remote response, the event loop in that worker did nothing else: no other request in that process moved. The database looked fast because it was idle, and the requests that needed it never got far enough to ask.
That is why the cheapest endpoint was the clearest symptom. It had no work of its own to hide behind, so what it measured was time spent waiting to be scheduled.
The fix was small: move the blocking client onto a bounded thread executor. The habit change was the real outcome. I now track event loop lag as a first-class metric right next to request latency, and I treat every third-party SDK as synchronous until I have read its client code instead of its README.
Per-request latency tells you how long one request took. It tells you nothing about whether the process could do anything else during that time. Those are different questions, and only the second one explains why unrelated endpoints degrade together.
I write these up in longer form here: https://polycratia.com/c/4e097522
If you run async services in production, I'd want to know what exposed the blocking call for you: loop lag, or a latency graph that stopped making sense...
- $ · permalink
The service I'd rewritten was correct. The rows in its table were not.
I had pulled one module out of an inherited monolith and rebuilt it as a standalone service. Clean state machine with explicit transitions, plus a test for every illegal jump. It ran about two weeks before records started showing up in states my own code could not produce.
The rewrite itself held up. What I had missed: the monolith still had an admin panel and a nightly job writing to those same tables directly, and both predated the module I replaced. My state machine was true only for traffic that came in through my service. Everything else wrote whatever it liked, and then my retry logic woke up and acted on that state as if it had produced it.
Nothing went down. What it cost me was a couple of weeks of not trusting my own invariants, which is probably the worse outcome: you start adding defensive branches instead of fixing the boundary.
So now, before any rewrite, I answer five things in writing first.
Who else writes to this data, outside the code I'm replacing?
What states already exist in production that the new code has to accept, not just produce?
Which callers depend on the current shape of the output?
What happens during the window when both versions write?
What's the rollback path once both of them have?
A rewrite replaces code. It does not replace the other authors of the system, and those are the ones who will quietly disprove your design.
I'm writing up more of these migration post-mortems over at https://polycratia.com/c/7fc186bc
If you've done a monolith-to-services split, I'd like to hear what turned out to still be writing to your tables after you thought you owned them.
- $ · permalink
The query plan never changed: three seconds became 200 milliseconds. The fix was to stop computing the number on read.
The endpoint returned a balance. Balances feel like stored data, but in a ledger they are a conclusion: sum every entry for the account, filter by state, subtract holds. That scan grows with account history, so response time becomes a function of how long someone has been a user. No index fixes that. You are paying for correctness on every request.
So the work moves to write time. The balance becomes a stored row, updated inside the same transaction that appends the entry. The read turns into a primary key lookup.
The cost is the honest part. A materialized number can drift: a backfill, a manual correction, a job that writes entries outside the main path. The pattern is only safe with three things. The entry and the update sit in one transaction, not two. The id or version of the last entry gets stamped on the materialized row, so you can tell what it reflects. And a recompute job re-derives from entries and alarms on a mismatch instead of quietly repairing it.
That last point is probably where most of these optimizations rot. Teams ship the cache and skip the audit, and a year later nobody can say whether the fast number is the true one.
I treat speed like this as a trade and not a win. You swap a per-request scan for an invariant you now defend forever. Worth it for balances, order book totals, catalog aggregates: numbers read far more often than they change. Not worth it for anything a human reconciles by hand.
I wrote up the drift-detection side in more detail here: https://polycratia.com/c/8a61f4a6
Where you draw that line is what I'd want to know: which numbers you still compute on read because you don't trust a stored one...
- $ · permalink
The parser kept running. Prices kept updating. Nothing errored, and nobody paged me.
A retail source had started redirecting requests to a different regional storefront, based on where it thought the client was sitting. My scraper followed that redirect without complaint. Same page structure, same selectors, numbers in exactly the shape I expected.
Different currency.
The bug was that a price in my pipeline was a number. Just a number. Currency lived somewhere upstream as a property of the source, configured once and then assumed forever. So the parser did its job perfectly and wrote confidently wrong rows for days.
The expense was not in the wrong prices. Everything downstream had already consumed them: delivery quotes, comparisons, what customers saw. Re-running the parser fixes the catalog, and it still does not tell you which decisions were made on the bad version, and it does not restore anyone's trust in the number on the page.
Two things changed in how I build parsers after that.
A price does not cross a boundary as a bare number anymore. It carries its currency and its minor-unit scale, and anything that receives it has to accept both or reject the row. If a value can be misread, I make the misreading impossible to express.
And the parser now asserts the identity of what it actually fetched before writing anything: final URL after redirects, locale, the currency marker on the page. A source is allowed to change its mind about who I am. It is not allowed to do that silently.
The cheap version of this check is a per-source sanity hold: if today's price distribution for a source shifts beyond a band, stop writing and flag it. It catches the whole family of bugs where the scrape succeeds and the meaning changes.
I write up more of these failure patterns from production systems here: https://polycratia.com/c/fa736460
If you run scrapers or third-party integrations at scale, the thing worth locating is probably the point in your pipeline where you would notice that a source quietly started answering a different question than the one you asked...
- $ · permalink
The list endpoint took 3 seconds. I ran the same query by hand and the rows came back in well under a tenth of that.
The rows were not the problem. The total was.
It was a filtered catalog listing on a marketplace: text match and facets, with a dozen optional filters hanging off it. Postgres served the page of twenty rows straight off an index. Then it ran a second query to count every matching row, so the UI could render "page 7 of 318". That count had no choice but to walk the entire filtered set, on every request, for a number sitting at the bottom of a screen.
I dropped the exact total and moved to keyset pagination. 200ms.
Then the moderation queue broke. Not visibly, since nothing errored. The compliance side had been reading the unmoderated count every morning as their workload signal, and now it showed them nothing useful. That was not written down as a requirement anywhere. It was a number on a screen that someone had quietly built a daily process around.
So the count went back, but only there: refreshed on a schedule, stored, and labelled with how stale it was.
What I do differently now is price a response field by field before I touch anything. Which part of this payload costs a full scan, and who actually reads it. An exact total on a filtered list is usually the most expensive field in the response and the least used one.
I write up more of these on the blog: https://polycratia.com/c/c2dd710e
What is the most expensive single field in your heaviest endpoint?
- $ · permalink
"It works on my machine" is rarely about the machine. It's a concurrency problem wearing an environment costume.
Containers fixed the half that was library versions and OS paths. The half that still breaks production: your laptop runs one request, one worker, one tab, so every code path gets serialized by the absence of a second actor.
Production has no such shortage.
A customer double-taps submit. A provider redelivers a webhook because your 200 came back slow. A queue redelivers a task after a visibility timeout and a second worker picks up the same row. Read-modify-write in application code is correct right up until something interleaves with you, and locally nothing does.
Arrival order is the other half. On your machine, events arrive in the order you created them. In production the success callback can land before the record it refers to exists, and a handler written as a straight line now has to tolerate arriving at step three first.
So I stop chasing parity and move the invariant to where concurrency actually gets arbitrated: a unique constraint on the natural idempotency key, state transitions enforced by a conditional update instead of an if-statement in the application. Then the default test runs the handler twice, and out of order.
Environment parity is cheap now. Contention parity is the one that rarely ships.
I write up more of these failure modes here: https://polycratia.com/c/2fa28b0f
In practice one of them costs you more, duplicate delivery or out-of-order delivery, and you probably already know which...
- $ · permalink
An endpoint that sits there for 3 seconds is usually not waiting on data. It is deciding, row by row, whether you are allowed to see any of it.
The mechanics are dull, and that is why they survive review. The handler loads a page of records, then asks an authorizer about each one: does this user own it, is their role scoped to this merchant, is the object in a state they may read. Each check costs a few milliseconds. Each one may touch Redis or a second table. Multiply that by a couple hundred rows and you have an N+1 inside the access layer instead of the serializer, and it hides well, because no single span is slow. The database looks healthy. The query log looks healthy. The time is scattered across thousands of operations too small to notice one at a time.
Pagination makes it worse. If filtering happens after the fetch, you read and authorize more rows than you return just to fill a page, so the invisible work scales with how much the user is not allowed to see.
When I redesigned the access model on a payment platform, I did not make the checks faster. I moved the decision into the query: resolve the caller's scope once per request into concrete predicates, push them into the WHERE clause, and let the database do the filtering it is built for. Per-row callbacks collapse into a set membership test.
The metric worth adding: rows examined per row returned. Once that ratio drifts above 1, your latency is a permissions problem wearing a performance costume.
I write up more of these debugging patterns on the blog: https://polycratia.com/c/fcba111e
So it is worth knowing which of the two your authorization runs in: the query, or a loop after it...
- $ · permalink
A borrower repaid in full, and the investor payouts came to one unit less than what came in.
One unit. On a single deal that is dust. The ledger filed it as an unmatched remainder and parked it, which is the only reason I ever saw it.
The cause was pro-rata rounding. Each investor's share of a repayment in a multi-investor deal has to be rounded to the smallest currency unit, and nothing in the code owned the gap between the sum of those rounded shares and the amount actually received. The split was computed, never reconciled. So on every repayment of every multi-investor deal, the remainder went somewhere: sometimes quietly absorbed, sometimes created out of nothing.
Nothing threw an error. To the person receiving it, every individual payout looked correct. The only visible symptom was a reconciliation queue growing slightly faster than it drained.
The missing unit was cheap. The expensive part was that for a stretch of time I could not prove the total, and proving it meant reconstructing splits from inputs I was lucky to have still retained.
What I do differently now: a distribution is not valid until the parts sum to the whole, and that assertion lives in the same transaction that writes the rows. The remainder gets an explicit owner, a deterministic rule for who absorbs it, stamped on the row instead of left to arithmetic. And money is integers in the smallest unit, not floats.
I write more about payment attribution and ledger invariants here: https://polycratia.com/c/eac71cd8
If you run a ledger that splits money between parties, the thing I would want to know is who owns your rounding remainder, and whether that rule is written down anywhere outside the code...
- $ · 9 min readWhy a DER length is checked against its parent, not the buffer
A length field is the first place two readers of the same certificate start to disagree. Before a single content byte gets examined, the length bytes already admit several readings: an indefinite…
- $ · permalink
The trace said 2.9 seconds. The database said 30 milliseconds. Between them sat one span, called "get token".
An internal payments endpoint had been parked at roughly three seconds for weeks. Nobody suspected auth. There was a token cache, I had written it, and it sat right there in the code.
It just never hit.
The cache key got assembled from the request context, and that context carried a value unique to every call. So each request minted a fresh access token from the OAuth service, and that service serialized mints. Under load, the queue in front of that one operation was the whole three seconds.
Latency was the cheap part of the bill. A token per request meant the credential store grew with traffic, revocation lists stopped meaning anything, and a short wobble in the OAuth service failed every request instead of the small share that genuinely needed a new credential. The slowness was just the symptom loud enough to notice.
Two changes stuck.
Hit rate ships with the cache now. Not inferred from the existence of a cache: emitted, graphed, alertable. A cache with no hit-rate number is an extra code path with optimistic naming.
And refresh is single-flight: many callers wait, one mint happens.
I write these up in longer form on the blog, if this failure mode looks familiar: https://polycratia.com/c/28cdd2f2
If you have a token cache in front of an internal service, you probably know the code is there. The hit rate today is the part I'd go look up...
- $ · permalink
The rewrite passed every test I had. Three days later support was sitting on a queue of people asking why they'd been told twice that their payment went through.
I was pulling a notification service out of an inherited monolith: fan-out to WhatsApp, Telegram, web and mobile push. The old code was ugly. One function queried a log table, assembled the recipient list, and sent everything inline. The new one was a consumer per channel, independent and retryable. Strictly better on paper.
What I missed: the old query joined against that log table before sending. Nobody had written that down as deduplication. Not in a ticket, not in a comment. It was a side effect of how the list got built, and it had been quietly absorbing retries for years.
My clean version had no such accident. Every retry was a new message. And on payment notifications a duplicate doesn't read as a bug to the person getting it, it reads as being charged twice.
So when people ask what to check before a rewrite, I don't start with architecture or service boundaries. I start here: which behaviors of this system are load-bearing without being specified anywhere? Then: who notices if they change, and how fast do they notice?
What I do differently now is run old and new side by side on real traffic before cutover, then diff the outputs. Every difference becomes an explicit decision: keep the quirk, or remove it on purpose. Nothing inherited silently, nothing lost silently.
I wrote up the rest of the questions I ask before touching an inherited system here: https://polycratia.com/c/feff745e
If you've done a rewrite, I'd like to hear which accidental behavior turned out to be load-bearing...
- $ · permalink
The repayment landed. Every investor got paid. The deal still would not close.
Multi-investor loan, pro-rata returns. The distribution code took the incoming repayment, multiplied it by each investor's share, rounded, and wrote the payouts. Each row was correct on its own. The sum was not the amount that had actually arrived.
The gap was smaller than the smallest unit anyone cared about. It still sat in the ledger as an unexplained delta on that deal, so reconciliation flagged it and refused to mark the repayment settled. One deal, then a handful. Eventually the people doing moderation stopped trusting the settled flag and started checking deals by hand.
That was the real cost. Nobody lost money. What got lost was the ability to believe a field, and a field you don't believe has to be replaced by a human.
The fix was not better rounding. It was treating distribution as splitting integer minor units instead of multiplying decimals: allocate the floors, then hand out the leftover units by largest remainder. The residual gets an explicit owner instead of quietly disappearing. Then check that the parts sum to the total inside the same transaction as the write, not in a nightly job afterwards.
Now I write that invariant before the business rule. Anywhere money fans out to several parties, I assume the shares will not sum, and I make the code prove they do.
I write more about this kind of production detail on my blog: https://polycratia.com/c/c037db37
If your system splits money across parties, someone owns the remainder: you should be able to name them...
- $ · permalink
Severity has almost nothing to do with how wrong a bug is. It is how many rows it wrote before anyone noticed.
Crashes are the cheap kind: they stop, they page someone, they leave a timestamp. The expensive kind produces plausible values, a price, a weight, a fee, a settled flag. Plausible passes validation, because validation checks shape and range, not truth. It passes monitoring too, because nothing is down. So it persists, and persistence is the whole mechanism that turns a one-line mistake into an incident.
Fan-out finishes the job. A derived value gets copied into a quote, an export, a document somebody already signed, a partner's cache you do not control. Each copy is a second source of truth that no longer agrees with anything, and none of them will tell you they are stale.
Then the asymmetry: the deploy fixes forward from the moment it ships. The data problem runs backwards from that moment to whenever the bug started, and usually nobody knows when that was, because the output looked fine the entire time.
So in review I don't only ask "is this correct". I ask "if this turns out to be wrong, how do I unwind it". Two cheap habits make the answer boring. Keep the raw input next to the derived value, so repair is a recompute and not a forensic reconstruction. Stamp derived rows with the version that produced them, so the affected set is a query instead of a guess.
A fix without a backfill path is half a fix. The half that stays in production is the data.
I go deeper into how I design for that in my notes here: https://polycratia.com/c/5f64ceca
Pick a derived value in your system and imagine it has been wrong for the last three months. Either you can select the affected rows today, or you are going to be reading logs...
- $ · permalink
On my laptop the migration ran clean in under a minute. In production it stopped partway through, on rows older than the code that was reading them.
I was moving an inherited service off Python 2. My local database was convenient in a way I had never examined: every row in it had been created by the new code. Text was text. Optional columns were populated, because my own write path populated them every time. The data agreed with my assumptions because my assumptions had written it.
Production data had history. Rows written years earlier by the old runtime. Rows imported by a parser that does not exist anymore. Rows from a schema version where that column was genuinely nullable. Some held bytes where the new path expected a decoded string.
The cost was not the crash itself. The job had no resume point, so it left part of the table in the new shape and part in the old, and the code meant to tolerate both shapes was the thing I was in the middle of deploying.
So now, before any migration that touches data, I start with a read-only census: null counts per column, which types and encodings are actually present, how far back the oldest writes go. Then I make the job resumable, idempotent per row, and able to quarantine a bad row instead of aborting the batch.
Fixtures I write myself only test my code against itself.
I keep longer notes on legacy modernization work like this on my blog: https://polycratia.com/c/e6282887
If you check one thing this week, check the oldest row in your main table, and which version of your code wrote it...
- $ · permalink
I cut the endpoint from 3 seconds to 200ms, and the app on the warehouse floor got slower.
I had split a fat order response into a thin list plus a detail call per line. The list query stopped joining everything at once, so the number on my dashboard dropped exactly as designed.
Then the operations people told me picking felt worse.
On a handheld over patchy wifi, the screen now needed one round trip per line item before it could render anything useful. Each call was fast. Together they weren't. Connection setup, auth, a retry whenever the link dropped, all of it multiplied by the number of lines on the order.
The rework is not what it cost me. It cost me credibility: the metric I reported had improved while the job people actually do had degraded, and I only found out because someone on the floor complained.
What I do differently now: the unit I measure is a user action, not an endpoint. "Courier opens an order", so total round trips, total bytes, worst case on a bad network. Endpoint latency sits inside that as one factor, not as the answer. And before I split a response, I count the calls the client will be forced to make afterwards.
Request latency is probably the easiest thing to improve, and the easiest thing to improve at someone else's expense.
I write up more of these trade-offs on my blog, if that's useful: https://polycratia.com/c/128a8657
So it's worth knowing where you draw your latency budget: per endpoint, or per screen...
- $ · permalink
Every test I wrote passed, and then the agent started describing a product feature that did not exist.
The pipeline adapts a retail catalog for another market: items come in from parsers, an LLM rewrites titles, attributes and descriptions, and the result goes back to the catalog.
In testing I fed it items I had picked by hand. Every field was populated. The job looked like translation...
In production the parser is the input, and a real parser returns partial items. A spec block that did not load. An attribute the source page rendered client-side. Nothing errored. The field just arrived empty.
My prompt asked for a complete, market-ready product record, so the model completed it. Material, compatibility, care instructions, written in fluent copy you cannot tell apart from the fields that were genuinely sourced.
That is what cost me. A crashing agent is a ticket. An agent that fills gaps produces output that passes human review, because nothing in the record marks which fields came from a supplier and which came from a model.
What I do differently now: input carries provenance per field, and the model may only emit fields that map to a source field that is present. Missing stays missing and gets routed to a human instead of written. Before anything touches the catalog, a validator compares the output field set against the input field set. That one check has caught more than any eval I had.
Testing told me the agent was good at the task. It told me nothing about what the agent does when the data underspecifies the task.
I write up more of these production failure modes here: https://polycratia.com/c/463cc554
If you run models over scraped or third-party data, I am curious whether you carry field-level provenance through the model call, or only validate the final shape.
- $ · permalink
Every evening the endpoint got slower, and the database graphs stayed flat.
Query time barely registered. The handler was nothing: pull a catalog page, serialize it, return. Still p95 climbed, and it climbed in a pattern that matched nothing I could point at. Deploys didn't explain it, and neither did traffic volume or one heavy tenant.
What it did match was client network quality.
I was serving a mobile app straight out of a synchronous worker pool. A worker is not done when it hands back the response object. It is done when the last byte reaches the socket. Catalog pages are fat. A phone on a weak connection took seconds to drain one, and for every one of those seconds a worker sat pinned there, doing no work, still holding a database connection it had stopped needing.
So slow clients quietly turned into server capacity. Get enough of them at once and the pool saturates, and then everything slows down, including endpoints that touch no database at all. It read like a database incident. It was not one.
What I do differently now: a buffering reverse proxy in front of any synchronous pool, so the proxy soaks up the response and frees the worker right away. Release the database connection before serialization, not after. And I watch time-to-first-byte apart from time-to-last-byte: the first one describes your code, the second describes the networks your users sit on.
I write these up at more length here: https://polycratia.com/c/a9a63986
Your API is probably holding a worker somewhere after the actual work is done...
- $ · permalink
A verification callback showed up before my own transaction had committed.
Fiat-to-crypto onramp, real-time KYC moderation. The flow was simple: create a verification session, hand the user off to the provider, wait for the result.
Locally it worked, because locally I was the one firing the callback by hand, in a terminal, whenever I felt like it. It arrived last.
In production the provider answered faster than my own write path. The callback hit a handler built to update a row that did not exist yet. It 404'd. The provider recorded a delivery attempt against an error and moved on.
So the provider counted those users as approved while my system still showed them pending. People re-uploaded documents. The moderation queue showed state that was no longer true. Nothing crashed, nothing alerted: the only symptom was a session that would never resolve.
What changed after that: webhook handlers create, they do not update. Idempotent upsert keyed by the provider's own identifier, raw payload persisted before anything interprets it, and the originating request reconciling with whatever is already there instead of assuming it writes first.
And in dev I now fire the callback before the request that caused it. That is the default case in my tests, not the odd one.
"It works on my machine" usually means: on my machine I am the one thing producing events, and I produce them in an order that suits me. Production has independent producers, each with its own latency.
I wrote more about how I structure these provider handoffs here: https://polycratia.com/c/4be57371
If you consume provider callbacks, your handler probably needs to cope with arriving first.
- $ · permalink
3 seconds to 200ms usually says nothing about speed. It says how often requests now take the fast path.
Most endpoints optimized this way end up bimodal: a cheap path that reads something already computed, and the original expensive path, unchanged, still sitting behind a cache miss, a cold key, an invalidation. The median moved because the mix moved. Nothing got faster.
That distinction matters, because the two failure modes are not equivalent.
A flat 3 seconds is slow but survivable. The load it puts on the database or the upstream provider is constant, and you have already capacity-planned for it.
A 200ms median with a surviving 3-second path concentrates that same work into the moments when the fast path is empty: right after a deploy, after a bulk catalog import rewrites half the keys, after a provider outage clears and every stale entry expires in the same minute. You traded steady slowness for a correlated stall, and the stall shows up exactly when the rest of the system is already degraded.
So before I call it a win, I measure three things separately. The cost of the slow path on its own. The share of traffic that still takes it. And whether the service stays up if that share goes to 100 percent for a few minutes.
If the answer to the third one is no, you don't have a 200ms endpoint. You have a 3-second endpoint with good weather.
The fix is unglamorous: single-flight on the expensive path, so a thousand simultaneous misses cost one computation instead of a thousand, and staggered expiry so keys don't die in formation.
I write up more of these trade-offs here: https://polycratia.com/c/e7cc6a9c
If you have shipped this kind of rewrite, I'd look at what your p99 does in the first minute after a deploy...
- $ · 8 min readDeleting == and << so a secret leak is a build error
A key sitting in memory leaks three ways, and every one of them compiles without a warning: a comparison that returns as soon as two bytes differ, a print that lands in a log, and the buffer still…
- $ · permalink
Six seconds on a verification endpoint. The query inside it took forty milliseconds, and nothing in the logs was red.
I had instrumented the handler and the database, and both looked healthy. The time was going into one outbound call to a KYC provider, and inside that call the client library was retrying. Two failed attempts, backoff between them, the third one succeeded. From my side it looked like a single successful request with a strange duration. No error, no alert, nothing to grep for.
The latency itself was cheap. What it cost me showed up elsewhere: users staring at a spinner resubmitted, and that created duplicate verification sessions, duplicate records landing in the moderation queue and duplicate checks against the provider, two sessions for the same person that a compliance operator then had to reconcile by hand. A latency problem had quietly become a data problem.
So now anything that leaves the process gets measured at the attempt level rather than the call level: attempts per request, duration per attempt, plus a timeout I choose instead of whatever the SDK defaults to. If the call can tolerate waiting, it leaves the request path entirely and becomes a job with a status the client polls.
A fast database tells you almost nothing about an endpoint that spends its time in a place you never counted.
I write up more production post-mortems like this one here: https://polycratia.com/c/bc2c664f
One thing I'm curious about: do you log retry attempts as separate events, or does your tracing collapse them into a single span?
- $ · permalink
"It works on my machine" is almost never about how your machine is configured. It is about the authority you hold on it.
Locally you are one principal with unlimited rights. Database owner. Long-lived credentials sitting in a dotfile. No egress policy. Every dependency reachable on loopback. Nothing in that code path has ever been denied anything, so the branch where permission gets refused has never run. Not in dev, not in review, not in tests.
Production hands that same code a narrower identity.
A role that reads the table but cannot touch the sequence.
A short-lived token minted hours before the request that needs it.
An egress allowlist that permits the payment provider and quietly drops the verification provider.
A replica that answers your SELECT and refuses your INSERT.
The dangerous part: partial authority does not fail the way missing config fails. Missing config fails at boot, loudly, in one place, before any user is involved. Partial authority fails mid-flow, on the second call, after you have already returned 200 and told the user their payment is accepted or their verification is submitted. In payments and KYC paths that is the write carrying the record of what happened, the one you need to reconcile against later.
So authority I treat as a test parameter, the same way I treat data shape and concurrency. Run locally as the production principal. Deny by default. Assert the refusal path explicitly, credential expiry included, not only credential absence. A permission error you have never watched execute is not handled code, it is untested code with optimistic comments.
I write up more of these failure shapes from production payment and compliance systems here: https://polycratia.com/c/1dd0f661
Where does scoped identity bite you first: storage writes, outbound egress, or token rotation?
- $ · permalink
In payments, the small bug with the widest blast radius has nothing to do with arithmetic: it's returning 200 to a webhook one line too early.
A 200 reads like politeness and works like a durability claim. You are telling the provider you hold the event, so it can stop holding it for you. That retry queue is, in practice, the only redundant copy of that event you will ever have, and you don't control it. You can't replay it past the retention window, and the ack deletes it.
So the ordering matters more than the handler does.
Die before the ack, and the event comes back. Die after the ack but before the write commits, and the event is gone from the world, with nothing in your logs saying so, because the handler returned successfully.
The same bug wears a second costume: enqueueing a background job inside a database transaction. The broker takes the message right away, the worker picks it up, the transaction rolls back, and the job runs against a row that never existed.
Both are one line in the wrong place. Both fail in the one direction you cannot reconstruct afterwards: missing things leave nothing to count.
So the handler has a single job, persisting the raw payload: provider id, body, received-at. Then return 200. Processing happens afterwards, out of my own storage, where it is replayable. Dispatch after the commit, not inside it.
A bug that loses money loudly gets fixed in a day. A bug that quietly narrows what you can still recover compounds for months.
I write up more payment-side failure modes like this here: https://polycratia.com/c/4a736a3f
Probably worth knowing where your webhook path acks, before or after the commit...
- $ · permalink
Going from three seconds to 200ms is almost never the code getting faster. The work left the request.
The large wins of that shape I've shipped were relocations rather than optimizations. The work went to one of three places.
Write time. A balance you compute by summing a ledger on every read becomes a maintained row, updated in the same transaction as the entry that changes it.
Background. Outbound work (a provider lookup, a document render, a notification fan-out) moves behind a queue, and the response hands you a handle instead of a result.
The client contract. Pagination, narrower payloads, fewer embedded relations: the endpoint stops answering questions the client didn't ask.
What rarely makes the headline is that each of those buys latency with an invariant you now have to maintain. A maintained balance can disagree with its ledger. A queued job can fail after you already returned 200. A narrowed payload means the client makes a second call that your new number didn't count.
So the 200ms is real, and it describes a different system with a different failure mode, not a faster version of the old one.
In money systems I only take that trade when the derived value has a recompute path: a job that rebuilds it from the source of truth and reports the delta instead of quietly correcting it. If you can't prove the two agree, you didn't speed up a read. You probably created a second source of truth.
I write up more of these trade-offs, which reads may be stale, which must be computed, and how the proof is built, on my blog: https://polycratia.com/c/ebc53da3
Worth knowing which reads in your system are allowed to be stale, and who actually made that call...
- $ · permalink
Every test I had, the matcher passed. In production it skipped payments without a word, and only ones that had arrived before a certain stretch of the past.
This was a reconciliation job: incoming bank transfers tagged only by requisites, matched against user obligations. I had rewritten the normalization step. Locally every fixture matched. Staging matched. Then the unmatched queue started filling up with old transfers that had reconciled fine for years.
The environments were identical. The data was not.
That requisites column had been written by three generations of my own code. Different trimming, different handling of separators, different ideas about what counts as the same reference. Each generation was correct when it shipped. My fixtures all came out of the newest writer, so my tests only ever saw the current dialect. The older dialects existed nowhere except production.
The cost was a day of staring at a matcher that was, in isolation, right.
What I do differently now: before I touch any read path, I ask how many writers have ever touched that table. Then I pull a sample of the genuinely distinct value shapes, anonymize them, and keep one fixture per shape, including the ones no live code can produce anymore. Historical dialects are part of the schema whether you documented them or not.
Local parity gets you the same runtime. It does not get you the same sediment.
I write up more of these production-only failure modes on my blog: https://polycratia.com/c/a300efcf
When you change a normalizer, I'd like to know how you find out which formats are already sitting in the table.
- $ · permalink
An endpoint that answers in milliseconds on its own and takes 3 seconds under load is not slow. It is serialized.
On a latency chart those two look the same, and they need opposite fixes.
Serialized means the work itself is fine, requests just take turns. In money systems the usual turnstile is one row: a balance, a running total, a per-user account that every request updates. A transaction grabs the lock, does its small bit of work, commits. The tenth concurrent request sits through nine turns before its own work even starts.
This hides well, because the tools you reach for all measure a request in isolation. EXPLAIN ANALYZE on that exact statement comes back instantly: it ran with nothing else competing for the row. The plan is optimal, the index is used, and no amount of query tuning will move the number. The cost does not sit in the plan. It sits in the queue in front of the plan.
The signal that splits the two cases is cheap: plot latency against concurrency instead of against payload size. Slow code degrades with input. Serialized code degrades with neighbours, so more concurrent callers push p95 up while the CPU stays bored.
The fix is structural rather than tactical. Stop mutating the aggregate on the write path: append immutable rows and derive the balance, or shard the hot row so contention spreads across many. A ledger that appends scales with hardware. A ledger that updates one row scales with nothing.
I put more of how I reason about contention in payment flows on my blog, here: https://polycratia.com/c/4c51a029
Worth knowing which row is the hottest in your system right now: a balance, a sequence, an idempotency table...
- $ · permalink
An agent that passes your tests has been tested on one thing: the run where every tool call came back fine.
More test cases will not close that. It is structural. In a clean environment the dependencies answer, the schemas line up, the provider is not rate-limiting you, so the failure branches do not execute and nobody wrote them.
Production hands them over on day one. A timeout on a call that actually committed. A 429 halfway through a fan-out. A stale read. A provider answering 200 with an empty body.
The mechanic that turns that into silent damage: in most agent setups a tool failure goes back to the model as text. The error is just another token in the context, and the model's job is to keep going plausibly. So it keeps going. It retries a write that was not idempotent, or swaps in an approximation, or calls the task done.
You have handed error handling to the one component that cannot be relied on to fail the same way twice.
What I do instead: raw failures do not reach the model. Deterministic code above the loop classifies every tool result as retryable, fatal, or degraded, and only that classification enters the context. Fatal stops the run with a structured outcome. Degraded gets recorded as degraded, not smoothed into prose.
The rule I hold to: the model may decide what to do next. It may never decide whether something worked.
I write up more about how I build these pipelines here: https://polycratia.com/c/97c3ce35
If you run agents against real systems, I'd like to hear which failure mode slipped past your tests first...
- $ · permalink
A slow API on top of a fast database is not a contradiction. That is the normal outcome of measuring execution and ignoring assembly.
The database answers exactly the question you asked, in milliseconds. Then the response still has to be built. Every row becomes an object. Every object walks a serializer. Nested relations expand, computed fields run, permission checks fire per field, per row. None of that shows up in a query log, because none of it is a query.
What usually fails is the shape of the response, not its speed. An endpoint that returns a whole catalog entry with its variants, media, and pricing history is cheap to query and expensive to emit. That payload grows with your data, not with your traffic, so the endpoint gets slower month over month while every single query stays fast. Put it behind a list view and the cost multiplies by page size.
The part that keeps this hidden is instrumentation asymmetry. Databases ship with timing tools and slow-query logs. Serialization ships with nothing.
So I log two numbers per response next to query time: rows hydrated and serialized bytes. The moment you can see those, the slow endpoints stop being mysterious. They are the ones emitting a megabyte so a client can render twelve fields.
And the fix is a contract rather than an optimization: explicit field selection, server-enforced page caps, no implicit nesting. A list endpoint and a detail endpoint are different endpoints, even when they describe the same object.
I write up more production trade-offs like this one here: https://polycratia.com/c/3f00b94c
Your heaviest endpoint spends its time somewhere: in the query, or after it...
- $ · permalink
A rewrite rarely fails on code. It fails on state.
Code is the part you can read: diff it, test it, delete it. What kills a rewrite is everything the old system knows that stayed out of the code entirely. It lives in rows, in cron jobs, in the habits of the people who operate it.
So, the five questions I want answered before anyone writes a line:
-
Who writes to this database besides the application? Legacy systems collect writers: an admin script, a nightly job, a support tool, sometimes a human with psql. Every writer you can't list is an invariant your new code doesn't know it has to preserve.
-
Which business rules exist only as data? Status values few people can enumerate, a config table that changes behaviour, rows hand-corrected by support during an incident. Rewriting the code preserves none of that.
-
Who reads your tables directly? Reporting, analytics, another service that was not supposed to. Change the schema and you break consumers missing from the architecture diagram.
-
Which guarantees come from the runtime rather than the design? Ordering, single-worker assumptions, a lock that works because there is exactly one process. Services multiply processes, and accidental serialization is the first thing to go.
-
How will you prove the two systems agree? If the answer is not a comparison you can run continuously during a dual-run, you don't have a migration plan. You have a launch date.
In money systems the fifth one is not optional. Reconciliation between old and new is the only thing that tells you the rewrite is correct, and it has to exist before cut-over, not after.
I write about this in more depth on the blog, and this one is the long version: https://polycratia.com/c/2a908e4c
If you've been through a rewrite, you probably found something writing to your database that had slipped off the list...
-
- $ · permalink
A moderation list endpoint on a payment platform took about three seconds, and the query log said nothing useful. The queries were small and indexed. The slow part was authorization.
The access model was the piece I trusted most. Roles, tiers, assignment scope: tests covered all of it. It just answered one question at a time, though: can this user see this object. So the endpoint loaded a page of cases, then asked that question per row, and each answer went off looking for role and assignment data of its own.
Two hundred rows meant hundreds of correct, well-tested checks inside one request. Fast everywhere, slow in aggregate.
The cost was worse than the number. Compliance moderators page through queues all day (that is the job), so the slowest screen in the product was the one people used most, and the obvious suspects (database, serialization, the provider calls) were all innocent...
I did not make the check faster. I made the same rules expressible as a filter, so the database returned only visible rows and the page stopped asking row by row. Getting to around 200ms was mostly deleting a loop.
What I do differently now: when I design an access model, every scope rule has to be stateable as a query predicate, not only as a yes/no on an object already loaded. Per-object checks stay on single-object endpoints, where they belong.
I wrote up how I approach permissions and payment-side data access here: https://polycratia.com/c/0d9d31a3
If you have hit this: did you push authorization into the query, or precompute visibility?
- $ · 8 min readWhy a money type should refuse the division operator
A rounding mode is an input to a calculation, and Python's binary operators have nowhere to put it. a / b calls type(a).__truediv__(a, b): two operands, one result, no keyword arguments. So a money…
- $ · 9 min readNo Patch Is Coming: PHP 7.4, 8.0 and 8.1 After PHP2Shell
On 21 October 2026 CISA issued an alert and CERT-EU published advisory 2026-018, covering two remote code execution flaws in PHP core: CVE-2026-109618 and CVE-2026-109619, both rated CVSS 9.2. An…
- $ · permalink
"Works on my machine" is usually not a claim about the machine. It's a claim about uptime.
Your local process lives for forty seconds. The production one lives for three weeks. A whole class of defects needs time before it shows up, and time is the one variable you rarely provision locally.
Token refresh is the clearest case. An access token with an hourly lifetime never expires inside a local run, so the refresh branch ends up the least-exercised code in the integration, and it's the first thing that runs unattended, without you watching. In payment integrations that branch often is the integration: the happy path is decoration.
Then you get the slow accumulators. A client that leaks one connection per failed request is correct on your machine and fatal after a week of uptime. A module-level cache loaded at import is fresh in a local run, and arbitrarily stale in a worker that hasn't restarted since the last config change. A retry schedule with long backoff rarely reaches its later attempts, because you stop the process first.
This isn't environment drift. A short-lived process just hides the bugs whose trigger is duration.
So I treat lifetime as a test parameter, not an environment property. Shorten TTLs in the test config until expiry lands inside the suite. Run workers long enough to cross at least one refresh and one reconnect. In-memory state you don't rebuild from a durable store is a bet that the process dies before it goes wrong.
Staging proves the code runs. Very little proves it keeps running.
I write up more of these production-shaped failure modes here: https://polycratia.com/c/c613f1ab
Which duration-triggered bug cost you the most: token expiry, pool exhaustion, or a stale in-process cache?
- $ · permalink
The shipping rates endpoint took seconds to answer. Every query behind it ran in milliseconds.
I spent most of a day profiling the wrong layer.
The handler was not doing database work. It was asking several carriers for a quote (rates, available services, delivery windows), one call after another, sequentially. Postgres sat idle. The request was just waiting on networks I did not control.
The part I took longer to accept: on that endpoint there was no fast path for me to build. A carrier having a slow morning set the response time for every user who got as far as checkout, however clean my code was.
So I stopped treating outbound calls as an implementation detail and started treating them as the budget. Now, before the handler exists, I write down every external call it will make and what the endpoint is allowed to spend in total. Then: fan out concurrently instead of in sequence, give each call a timeout well under the endpoint budget, cache quotes keyed on normalized inputs, and return a partial answer that is explicitly labelled partial instead of holding the whole response hostage to the slowest participant.
A fast database buys you very little when the handler spends its life waiting on someone else's network. You cannot optimize a third party. You can only bound it, and decide in advance what the user sees once the bound gets hit.
I wrote more about these trade-offs here: https://polycratia.com/c/dd65261b
So when a third-party call blows your latency budget, the thing to settle early is what you return: stale, partial, or an error.
- $ · permalink
The migration looked mechanical. Python 2 to 3, a well-understood path, a codebase I had already read end to end.
Then I hit a function that normalized incoming payment requisites before matching. Nothing special: a few string operations, a lookup, a fallback branch. Under Python 3 it behaved differently on a narrow slice of inputs, and that slice lived only in real production data. Not in the fixtures. Not in anything anyone had written down.
That branch was not a bug. It was a requirement. Someone had encoded a quirk of how a specific payment channel formatted references, years earlier, and the only record of that decision was the code path itself.
The fix was cheap. The week after it was not: I spent that week finding out how many more of those existed. Every unmatched payment after the cutover arrived with the same question attached. Did I break this, or was it always like that?
So now, before I agree to any rewrite, my first question is not about architecture or stack or timeline. It is this: can the current system be observed well enough to know when the new one disagrees with it?
If the answer is no, the rewrite is not a rewrite. It is a guess that gets deployed.
What I do differently: I run both paths against real traffic before anything switches. Same inputs, both implementations, outputs compared, every divergence logged. No cutover until the divergence list is empty or explained. It is slower. It also turns "I think it works" into something you can show a client.
The uncomfortable part is that this reorders the usual five questions. People start with scope, cost, team, timeline, stack. Those matter. They also sit downstream of whether you can prove equivalence. A rewrite you cannot verify is a rewrite you cannot finish: you can only abandon it, or ship it and find out from users.
The same logic applies to monolith-to-services work, which is where I have spent a lot of the last ten years. Pulling a service out is easy. Proving the extracted service makes the same decisions as the code it replaced, on the inputs that actually occur, is the whole job.
I wrote more about how I approach inherited systems and migrations here: https://polycratia.com/c/b4c6f625
If you have done a migration on a system with money in it, I want to know what finally exposed the undocumented behavior for you: shadow traffic, reconciliation mismatches, or a support ticket...
- $ · permalink
The agent passed every run I threw at it. Then it hit production and I couldn't answer a basic question: which output did I actually publish?
The work was an LLM pipeline adapting product catalogs for other markets. In testing I pushed records through one at a time, read the output, moved on. In production that same step ran as a queued fan-out over thousands of records.
Queues retry. That's their job. A worker stalls mid-record, the broker redelivers, the step runs again.
What my test runs hid from me: a model call is not a pure function. Same record, same prompt, different text. A retry stops being a retry: it becomes a second opinion. So some records got written twice with two different versions, last write won, and nothing in the logs admitted a second call had happened.
Nothing crashed. Nothing alerted. I found it by diffing published catalog text against what I expected to be sitting there.
The cost was mostly trust. Before I could ship anything else through that pipeline, I had to work out which records had been generated more than once.
What I do differently now: I treat a model call as a side effect with an identity, not as a computation. Before the call I derive a key from the source record version, the prompt version and the model id. If that key already has a stored result, the retry hands back the stored text and never reaches the model. The key sits beside the output, so any row can be replayed and diffed months later.
A prompt that survives manual review is not a pipeline. A better prompt rarely closes that gap: what's missing is an identity for every generation you can look up.
I wrote up more on how I structure these pipelines (input contracts, keyed outputs, replay) here: https://polycratia.com/c/2d2b4dac
If you're running agents behind a queue, the identity of a model call either lives somewhere you can point at, or a redelivery quietly generates a fresh answer...
- $ · permalink
Seconds to load on the marketplace list page, and the first two things I tried were the wrong layer.
It was a P2P lending order book. Every row had to show how much of a loan was still open, and where each investor's pro-rata share stood against it. The endpoint computed all of that live: walk the investments for each deal, sum them, divide, return.
So I did what you do. Read the query plan. Added indexes. Shaved a little off.
Then I reached for a cache and stopped. These numbers are money. A stale remaining amount means two investors can both be told there is room for the same funds. The latency was annoying; that bug would have been expensive, and it would have surfaced as a reconciliation problem weeks later, far away from the endpoint that caused it.
The real change: stop asking the question at read time. The remaining amount became stored state, updated inside the same transaction that records an investment. The read turned into a lookup. Fast, and correct by construction instead of correct by recomputation.
What it cost me was a day spent optimizing an aggregation that had no business being on the read path at all.
What I do differently now: when an endpoint is slow, before I touch the queries I ask whether it is deriving something that only ever changes on write. If it is, the answer is rarely a faster query and almost never a cache.
I write these up in longer form on my blog: https://polycratia.com/c/627d4c37
Somewhere in your read path, I'd guess, something still gets recomputed that a write already knew...
- $ · permalink
Partway through a monolith-to-services rewrite, I found out the cut-over date was not mine to pick.
The system I inherited had quietly grown an integration surface. Payment provider callbacks. Carrier webhooks. A couple of partner endpoints. Those addresses all lived in someone else's dashboard, and some were set up years earlier by people who had since left both sides of the integration.
Each of those URLs was a contract I could not version. The handler behind it I could rewrite in an afternoon. Asking a provider to change a notification address on my sprint schedule was a different thing: their change windows ran on their own calendar, and one of them wanted a support ticket plus a round of sandbox re-testing before anything moved.
So it stopped being an architecture exercise and turned into a mapping exercise. Which of my endpoints does an external party hold the address of? What happens to a callback that lands mid-switch: does it get retried, or does it just disappear into a 404 that nobody owns?
The time went to assuming the system boundary sat where my code ended. It sat where someone else's configuration pointed.
Now I inventory the externally-addressed surfaces before any rewrite, and I keep the old paths alive as thin forwarders. No cut-over moment, no scheduling calls with third parties: the old URL keeps working while the logic moves behind it.
I've written up more of how I sequence migrations like this on my blog: https://polycratia.com/c/89136140
If you've done a rewrite with live third-party callbacks pointing at the old system, I'd be curious whether you forwarded or coordinated a window.
- $ · permalink
Locally the endpoint was fast, and in production the median looked fine. Then a slice of requests hit a wall at almost exactly three seconds, every time.
That shape cost me two days. A flat ceiling, not a spread, and I spent those two days looking in the wrong place. I profiled the handler. Checked the queries. Timed the serializer. All of it came back fine.
The three seconds was a retry. Not mine.
The HTTP client wrapping a payment provider integration shipped with a default retry policy: on a connection-level error, try again, twice, with a short sleep between attempts. The provider was failing on a small share of calls, and the library absorbed that quietly. One slow request is what my metrics saw. What actually happened was three outbound calls and two swallowed errors.
The latency was the cheap part of that bug.
Those retries were going out on a call path that was not idempotent. A duplicate charge attempt is a much worse problem than a slow endpoint, and nothing in the logs would have told me it was happening: from the application's point of view, one call went out and one result came back.
So two things changed in how I work.
I read the defaults of every client library that touches money or crosses a network boundary, retries and timeouts and connection reuse included, and I set them explicitly instead of inheriting them. A default I did not choose is still my behaviour in production.
And I log attempt counts, not just request duration. Duration tells you something was slow. Attempt count tells you why, and it is the only signal that makes a swallowed error visible at all.
The heuristic I kept: a timing anomaly with a flat ceiling is almost never computation. Computation varies. Waiting and retrying do not, they land on round numbers, because someone picked that number for you.
I write up more of these production post-mortems in longer form here: https://polycratia.com/c/66ff2bb6
If you have found one of these in your own stack, I'd be curious which exposed it first for you: attempt-level logging, or a duplicate showing up downstream...
- $ · permalink
"It works on my machine" is almost never about the machine. It is about concurrency of one.
Locally you are the only caller. One process, one worker, no retries, no provider hitting your webhook twice because the first response timed out, no second container holding a lock the first one thinks it owns. Any bug that needs two things to happen at the same moment is invisible by construction: not undetected, undetectable.
The shape is usually check-then-act. The handler reads a payment, sees it has not been credited, writes the ledger entry. Serially that is correct, and it stays correct through every local run you do. Put two concurrent callbacks on it and both reads finish before either write lands, so the balance moves twice. A process-level lock tells the same story: fine while there is one process, quietly a no-op the moment you run two replicas. Nothing in your environment can warn you, because your environment never had a second replica.
So I stopped treating this as an environment-parity problem. I push the invariant down to where concurrency cannot route around it: a unique constraint on the idempotency key instead of a lookup, conditional updates that fail when the state has already moved, state machines that reject an illegal transition rather than code that checks first and trusts the gap. Containers give you identical dependencies. They do not give you identical arrival order.
So the parity I go after is the concurrency level rather than the environment...
I write up more of these payment-side failure modes here: https://polycratia.com/c/f011b2ad
Your first real one probably surfaced in one of the familiar places: duplicate provider callbacks, parallel retries, or a lock that stopped working the day you scaled out.
- $ · permalink
If you run PHP 7.4, 8.0 or 8.1, no patch is coming for this one. Not late. Never.
On 21-23 October two critical remote code execution flaws landed in PHP core: CVE-2026-109618 and CVE-2026-109619, CVSS 9.2. Unauthenticated code execution on anything web-facing: mod_php, PHP-FPM, php-cgi, FrankenPHP, LiteSpeed. CISA and CERT-EU both say patch immediately, and exploit attempts are already being reported.
Emergency fixes went to the supported branches only: 8.5.12, 8.4.27, 8.3.36, 8.2.35. Anything older gets nothing.
That gap matters more than the CVSS score does. A bug has a fix date. An unsupported branch has no fix at all, so this stops being a patching job and turns into a stack decision.
Three things worth doing today if you stay on PHP:
Move to a patched 8.2+ build.
Set session.upload_progress.enabled=0 in php.ini, the only mitigation for the third flaw, CVE-2026-115560, disclosed 23 October with no patch yet.
Check what your hosting vendor has actually patched, not what the status page claims.
And if the honest answer is that this service should come off PHP entirely: send it to me today and within one day I will tell you what moving it to a supported, modern stack costs and how many weeks it takes. Small services come off in days, not months. I have done legacy rewrites before.
Message me.
- $ · permalink
A rounding remainder is the cheapest bug in a money system, and the one that scales worst.
One cent. No exception, no alert, no failed request.
The mechanics are what make it dangerous. Split a repayment across multiple investors pro rata and every line rounds correctly on its own, while the sum of the lines no longer equals the amount that actually arrived. The code is locally right and globally wrong, and that is exactly the shape of bug a unit test misses: the assertion you write is about one share, not about conservation.
Then it compounds. The remainder has to land somewhere, so it lands on whoever the loop happened to touch last. If the allocation iterates over an unordered query, "last" changes between runs, and the moment a payout gets retried the same input produces a different distribution. By then you don't have a rounding bug. You have an idempotency bug wearing a rounding bug's clothes.
So I treat distribution as a conservation problem before an arithmetic one: integer minor units, deterministic ordering, an explicit remainder policy, and one invariant checked at the boundary. The sum of allocations equals the source amount, or the distribution refuses to commit at all.
A balance that is off by a cent is not a small error. It is a ledger you can no longer prove.
I write up more of these failure modes on my blog: https://polycratia.com/c/34a9acd6
Worth knowing where your system puts the remainder today, and whether you could re-derive the same split twice...
- $ · permalink
A fast database is one of the main reasons an API stays slow.
No paradox there. Slow-query logs are duration instruments: they catch the one call over a threshold and stay blind to the thousand under it. When every query comes back in single-digit milliseconds, nothing crosses the line, so the cost moves out of duration and into count, where nothing is watching.
The mechanics are arithmetic. Request latency is per-call cost times call count, plus whatever assembly happens around it. Fast storage shrinks the first term and makes the second one invisible. A catalog endpoint that fans out one query per item, plus one per price, plus one per delivery option, can burn most of a request budget inside a database whose own dashboard reads green at p99.
It repeats a layer up: serializers that touch related objects, permission checks resolved per row, a cache lookup per entity over a network hop. Each one is cheap. None of them gets logged. All of them are serial.
So I treat queries-per-request as a first-class metric next to latency: counted per endpoint, attached to the request log, given a budget that fails loudly in tests when it drifts. Probably the cheapest instrument I know for catching a regression no duration threshold will report.
I write more about where request latency actually hides over on my blog: https://polycratia.com/c/50347654
Worth knowing where your own query count drifts fastest, in serializers, in permission checks, or in those per-row cache lookups...
- $ · 8 min readWhere a deposit watcher starts reading is not the reorg window
A deposit watcher carries two numbers that look interchangeable and are not. The confirmation depth decides when an incoming payment gets credited. The reorg window decides how long that credit can…
- $ · permalink
The smallest bug in a money system is a rounding call that nobody owns.
Split a repayment across several investors by share and you get fractions. Round each share to the currency's minor unit and the parts stop summing to the whole. One unit is left over, on every split, not once in a while.
It ships because nothing looks wrong. Every row is correct to two decimals. No exception, no failed request, no alert. The defect lives in the aggregate, and there is no single record you can point at and call incorrect.
Then the remainder goes somewhere. Whichever write lands last absorbs it, or it gets folded into a balance update, or it quietly disappears. A few thousand distributions later your ledger is internally consistent and externally wrong, and the first real signal is a reconciliation break far downstream, nowhere near the line that caused it. By then you are not asking how to fix the allocator: you are asking how to restate history.
So I treat distribution as a conservation operation rather than arithmetic. Amounts in integer minor units, never floats. Allocation of the leftover unit is deterministic, with an explicit tie-break, so the remainder has a named owner instead of a lucky one. Then an assertion before commit: the sum of the parts must equal the input, or the transaction fails loudly.
That last check carries most of the weight. Money code should be structurally unable to lose a unit in silence, because the cheapest place to catch drift is the function that created it: every layer after that only makes it more expensive to prove what happened.
I write more about designing invariants like this on my blog: https://polycratia.com/c/1c83664a
Where you enforce conservation is the part I keep going back and forth on: inside the allocator, or at the ledger boundary...
- $ · permalink
The rewrite stalled on a column called status.
The code was not the problem. An inherited Python 2 monolith with money flowing through it, pulled apart service by service: unremarkable work.
Then the new service started answering confidently and wrongly about old records.
Those status values had not meant the same thing across the life of that table. Early rows came from code paths deleted years earlier. Later rows reused some of the same values for a different state. Nothing in the repository recorded the switch, so the only place that history survived was the data itself.
Porting the module turned out to be the cheap half. The backfill is where the time went: I reconstructed era boundaries from the rows, worked out what each legacy value meant inside its own period, then ran old and new side by side against historical records rather than fresh traffic. On fresh traffic the two systems agreed from day one. Agreement on new rows told me nothing.
So the question I now put ahead of every other one before a rewrite has nothing to do with the code: can the new system interpret the oldest data as faithfully as the code that wrote it?
In practice that means inventorying each state field's value set by time window, and treating any value no current code path can still produce as a separate legacy meaning until proven otherwise.
I keep the longer version of this, the full set of questions I run before agreeing to a rewrite, on my blog: https://polycratia.com/c/281aae59
If you are weighing one right now, you probably want to know which of your status values your current code can still write, and what the rest used to mean.
- $ · permalink
The parser ran clean on my laptop: full catalog, prices, stock flags, all of it.
The same commit on the server gave back a catalog too. Just not the same one.
Prices were off by a few percent. Some items were missing. A few carried availability states I had never seen locally. Nothing crashed, nothing logged an error. The scraper did exactly what I wrote it to do, against a page the retailer had chosen to serve a datacenter IP rather than a home connection.
That site shaped its own response: region, currency, language, availability, and sometimes the markup itself. My machine was not a cleaner version of production. It was a different customer.
It cost me days of debugging the wrong layer (selectors, parsing, encodings) because the code was innocent and the input was quietly lying.
What I changed after that, and still do on every catalog pipeline:
I treat the environment as an input, not as ambient context. Locale, currency, declared region, request headers, egress path: pinned explicitly and asserted on, not inherited from whichever machine happens to run the job.
I keep the raw response that produced a parse, not only the parsed result. When the numbers look wrong I want to compare bytes, not theories.
And I diff each parse against the one before it. Real catalog data drifts slowly: a page served under different assumptions drifts all at once.
So "works on my machine" is seldom about the code. It is about which hidden inputs your machine supplied for free, and which ones production refuses to.
I write longer versions of these production notes on my blog, if that is your kind of reading: https://polycratia.com/c/223c85fe
If you run scrapers or catalog pipelines, something probably exposed your environment difference in the end: a data mismatch, a customer complaint, or a reconciliation that would not add up. I'm curious which one it was for you...
- $ · permalink
The catalog list endpoint answered in about three seconds, and I spent the first stretch of that reading query plans.
Indexes first. Then prefetching, then a cache sitting in front of the ORM. Each one helped a little, and none of them moved the number, because the time was never going to the database.
That endpoint returned products for a cross-border storefront, and every row carried a shipping estimate. The estimate came out of carrier integrations: a rate lookup, per destination, per weight. A page of results meant a page's worth of outbound calls, sequential by accident, each one fast enough to look innocent on its own.
What changed in the end was the promise the response made, not the code.
Shipping estimates left the list. Rates got precomputed per destination zone and weight bucket, refreshed on their own schedule, and the exact quote moved to the item view, the one place where a buyer is actually about to commit. The list went back to reading its own tables and answered in a couple hundred milliseconds.
The honest version of that result: I did not make anything faster. I stopped doing work inside a request that never belonged there.
Since then, my first question on a slow list endpoint is not which query is heavy. I ask which fields get computed per row, and whether any of them depend on someone else's uptime. If one does, it is not a field. It is a separate call.
I write up more of how I approach this kind of work on my blog: https://polycratia.com/c/5de6bede
I would like to hear which per-row computation you ended up pulling out of a list response...
- $ · permalink
Support told me a screen in the moderation panel took about three seconds to load. I profiled the endpoint behind it and the handler came back fast: query, serialization, all of it well inside budget.
The time was not in the endpoint. It was in everything that ran before it.
Every request to that payment platform re-validated the bearer token against the OAuth service, then expanded the caller's roles into an effective permission set, a walk across the access model, once per request. And the panel did not make one request per screen. It made a dozen in parallel. So one page load meant a dozen introspection round trips and a dozen permission expansions against a service nobody had sized for that.
The endpoint I was asked to optimize was innocent. The slow part had no owner, because nobody thought of it as part of the request at all.
Two things changed after that.
I measure request time in two halves (before the handler, and inside it), so middleware cost stays visible instead of folded into one number.
And authorization state gets computed when a token is issued, not when it is used. The request path reads a materialized permission set and checks a revocation list. Nothing more.
Access control is request-path code. If it does not show up in your timings at all, that does not make it cheap: it makes it unmeasured.
I write these up in longer form on my blog: https://polycratia.com/c/8aec660e
Curious how others settle this one: you cache token introspection, or you push permissions into the token and live with the revocation lag...
- $ · permalink
Slow APIs stay unexplained mostly because the database is fast.
Every query comes back in single-digit milliseconds, nothing trips the slow-query threshold, the data layer dashboard is green, and the endpoint is still visibly slow.
Here latency is a count, not a cost: the number of sequential round trips a request makes before it can answer.
The mechanics are mundane. A list endpoint loads its rows and then enriches each one: a lookup per item, a permission check per item, a counter per item. Each call is cheap, and each call waits for the one before it. Storage is fast, so no single hop ever looks wrong, and every tool in the stack measures duration per query instead of depth per request. The request pays for depth; the monitoring reports duration. That mismatch is the whole blind spot.
What I do about it: I treat round trips per request as a first-class number, the way you'd treat p95. Log the count per endpoint, not just the time. Then watch one property, whether that count stays constant or grows with the size of the result set. A handler whose query count scales with how many items it returns is not a performance bug you optimise away. It is a shape you have to change: one batched read instead of per-item reads, or enrichment moved out of the hot path, or the value precomputed at write time rather than assembled at read time.
The uncomfortable part: fast storage buys you room to be wasteful, and that room is exactly where the latency piles up. Teams that migrate to faster hardware and see nothing improve usually land here. They made each hop cheaper without making the request shallower.
I wrote the longer version of this, including how I split request timing so the trips stop hiding behind each other, here: https://polycratia.com/c/858b526b
If you've chased this before, I'd be curious: query count per request constant, or quietly scaling with page size...
- $ · permalink
Pass the tests and the agent has proven one thing: it can reason about a world that held still while it thought.
Production does not hold still.
The failure lives in the gap between plan time and act time. The agent reads state, spends seconds or minutes forming a plan, then acts, and by then the payment settled, the balance moved, the document picked up another signature, the catalog item was repriced. The reasoning was correct for a state that no longer exists. Evals miss it, because a fixture is frozen by definition.
Three things change the outcome, and none of them are about the model.
Every read the agent is allowed to act on carries a version. If the version moved, the write gets rejected, and the agent re-plans instead of overwriting someone else's truth.
Idempotency keys come from the intent, not from the attempt. A retried agent step converges on one effect rather than two: the same discipline payment rails have needed since long before agents existed.
Each action re-validates its own precondition at execution time instead of trusting the plan that produced it. The plan is a proposal. The action layer is the authority.
The real shift is in how you read the thing: an agent is a slow, concurrent, retrying client of your system, so it needs the guarantees you would give any other such client. Most production agent failures I see are ordinary concurrency bugs wearing a new costume.
More on this on the blog: https://polycratia.com/c/0fa23d3a
So if you run agents against live state, you are doing one of two things: versioning the reads, or shortening the window and hoping...
- $ · permalink
Most rewrite decisions get made by reading the code. That is the wrong evidence: code tells you what offends you, not what the system actually does in production.
Five questions I ask before anything gets rewritten.
Can you describe the current behaviour precisely enough to replicate it? Not the intended behaviour, the real one: the rounding quirk, the retry that fires twice, the status that only ever shows up for one legacy payment channel. In inherited systems those quirks are load-bearing. If nobody can list them, what you have is not a rewrite, it is a guess with a deadline.
What is the smallest piece you can cut out and run in parallel? Monolith to services work goes well when the first slice is one bounded capability, say a file service, or a notification fan-out, or a parser, with a seam you can route traffic through. It goes badly when the first slice is "the core".
Can both versions run at once on the same data? If not, you have signed up for a cutover weekend instead of a migration. Parallel running is what turns opinion into evidence: same input, two outputs, diff them until the diff is boring.
What is the rollback? Python 2 to 3 migrations taught me this one bluntly. Library by library, deployable at each step, reversible at each step, because the failure rarely arrives in the module you were editing. It arrives in a Celery task that only runs at month end.
Is the pain in the code or in the coupling? Ugly code inside clear boundaries is cheap to live with. Clean code sitting on a shared mutable database with no contracts is what actually blocks you. A rewrite that preserves the coupling just relocates the problem into a newer framework.
My take: the question is rarely "rewrite or not". It is whether the system is observable enough to be replaced incrementally. If it is, you rarely need a big rewrite, since the increments get you there. If it is not, then instrumentation is the first project and not the second.
I wrote up how I sequence this, seams, parallel running, and what to measure before you touch anything, here: https://polycratia.com/c/038bc566
If you have done one of these, I'd like to hear what finally told you the old system was safe to turn off.
- $ · permalink
I ran a catalog translation pipeline over a handful of products, read every output myself, and shipped it.
Then it ran over the full catalog, and I had no idea whether it was working.
Nothing crashed. Every item came back well-formed: fields populated, language correct, no exceptions, no retries. The pipeline called all of it a success. Then a later sample showed units quietly converted, a model code rewritten as if it were an ordinary word, and attribute keys that no longer matched the ones search and delivery depended on.
The model was not where this broke. In testing I was the grader: I read each output and decided whether it was right. In production nothing played that part, so a plausible answer and a correct answer looked the same to the system.
What I do differently now: every step emits a verdict that something outside the model can check. Invariants pulled from the source (numbers, units, identifiers, the set of attribute keys) have to survive the transform, and an item that breaks one gets parked instead of published. The agent is allowed to be wrong. It is not allowed to be wrong silently.
I write up more of these production failure modes on my blog: https://polycratia.com/c/b56d3b37
If you run agents at scale, what catches a bad output in your pipeline before a user does?
- $ · permalink
A fast database and a slow API do not contradict each other. Usually it's the same story: your query time is fine, your connection acquisition time is not.
The mechanic goes like this.
A request checks out a database connection, and somewhere inside that scope it calls someone else's system: a carrier rate lookup, a KYC provider, a payment gateway. The query itself has already finished. The connection stays checked out anyway, doing nothing but waiting on a socket you don't control.
So the pool becomes a budget, and an upstream you don't own is spending it. One carrier gets slow and every endpoint in the service degrades, including the ones that never touch that carrier. The database stays fast the whole time, and your query metrics keep saying so, which is why the investigation lands in the wrong place.
What I do about it:
Measure pool acquisition wait as its own number, separate from query execution. If you only have one latency metric for the database, you can't see this at all.
Don't hold a transaction open across an external HTTP call. Commit before, or defer the call after, but don't straddle the two.
Give every third-party client a timeout shorter than the pool's wait, plus its own concurrency limit, so a degraded upstream starves itself instead of the shared pool.
The general rule: latency accrues to whatever resource is held longest, not to whatever is slowest.
I went deeper on how this plays out in payment and delivery integrations on the blog: https://polycratia.com/c/6b118266
Worth knowing where your pool gets drained first: an external provider, or something internal you had stopped thinking of as a network call...
- $ · 8 min readA testnet xpub under a mainnet encoder: addresses nobody scans
A deposit service pairs an extended public key with an address encoder, and there is exactly one way to get that pair wrong that produces no error at any layer. A testnet key under a mainnet encoder…
- $ · permalink
Your database can be fast and your API still slow. All a fast database proves is that one of the four things in your request path is cheap.
Query time gets measured per statement. Request time is every statement added up, plus every gap between them. Thirty queries at two milliseconds each is sixty milliseconds of pure waiting before a single byte gets serialized, and each one of those statements looks healthy in the slow query log. Nothing there is slow. The request still is.
Then there is the time before the query runs at all. Connection checkout queues behind whoever is holding a pool slot, and the classic holder is a transaction left open across an outbound HTTP call. Database idle, pool empty, and the latency shows up nowhere in database metrics: it happened outside the database.
And there is the time after the rows arrive: materializing objects, filtering permissions row by row, serializing nested payloads. That is application CPU, and in a worker process it blocks everything else on that worker.
So the fourth thing is the response contract itself. An endpoint with no bound on what it returns gets slower as your largest account grows, and no index fixes that.
What I instrument per request: query count, time inside the database, connection wait, serialization time. Four numbers, and the guilty one usually stands out.
I write up more of these production trade-offs here: https://polycratia.com/c/a0912ff4
Next time you chase a slow endpoint, check which of the four actually owns the latency...
- $ · permalink
Your agent didn't get dumber in production. It just stopped being the only thing touching the data.
In testing an agent runs alone. It reads state, thinks for a few seconds, calls a tool, and the world it read is still the world it writes to. Production puts a gap in there, and the gap is where things break. Between the read and the write a payment settles, a listing sells, an operator makes a call in the admin panel, another worker picks up the same job.
So the plan is sound, the tool call succeeds, and all of it lands against a version of reality that no longer exists. It doesn't present as a model failure. That's why teams go looking for better prompts.
I treat an agent loop the way I treat a payment flow: no action on a read that isn't proven current. Conditional writes against the version the agent carried in, never a blind update. Scope the actions so the state a decision depended on is the state it locks. And keep an explicit line between an action that can be retried and a decision that has to be re-made, because an agent that "retries" usually re-plans from fresh context instead of resuming. That is a different operation, and in anything that moves money or inventory it needs to be treated as one.
Test harnesses hide all of this: they're single-actor by construction. Nothing competes with the agent unless you build something that does. Concurrency is a condition you have to inject on purpose, the same way you inject a timeout or a malformed payload.
I've written more on where these pipelines actually break in production, over on the blog: https://polycratia.com/c/3a5d4b9d
If you're running agents against live state, you probably have a staleness window between read and act, and it's worth knowing whether anything in the system enforces it...
- $ · permalink
The most expensive bug in a payment system is usually off by one cent.
A crash you fix in an hour. A rounding rule that quietly hands out 99.99 instead of 100.00 runs for months, because nothing throws and every single record looks plausible on its own.
The mechanics are dull. You split an amount across several recipients: pro-rata returns to multiple investors in one deal, or fees across parties, or a refund across the original payment legs. That split is a division, and it rarely lands on whole units. Each share gets rounded on its own, so the sum of the rounded shares is no longer the amount you started with. A cent sits unassigned, or a cent gets invented.
Nothing in the request path objects. Rows are written, balances move, and the user sees a credible number. The error surfaces only where something finally checks a conservation invariant: reconciliation against what the bank actually settled. By then the residue is smeared across thousands of records, and a rounding artifact looks the same as a genuine mismatch or a transfer that really did fail. You stop trusting the whole report, not just the bad rows.
So I treat the split as a first-class operation instead of an arithmetic detail. Amounts in minor units as integers, never floats. Shares derived from the total, then the remainder assigned by one explicit, documented rule: largest share, or deterministic order, or whatever the business agrees to, as long as it is written down and tested. And the invariant asserted inside the same transaction as the write: the parts sum to the whole, or nothing is written.
That last line is the actual lesson. A small bug grows into a large production problem when the system has nowhere it is allowed to refuse.
I write up more of these production notes here: https://polycratia.com/c/1e0e5feb
I'd want to know where your system asserts its money invariants: at write time, or in a nightly job...
- $ · permalink
The migration was meant to be mechanical. Python 2 to 3, a codebase someone else wrote years earlier, a test suite that passed.
It passed because it tested what the original authors knew they had built. About what the system had quietly learned to do in production over the years it ran, it said nothing.
The parts that hurt were not syntax. They were behaviours that had not been written down: a string that was sometimes bytes depending on which path produced it, ordering that downstream code depended on without declaring it, an edge case someone patched once and left undocumented because that week the patch was clear to them.
The diff does not show any of that. Production shows all of it.
The cost was time spent in the wrong place. I had scoped the work as a language migration, then found myself doing archaeology: reading logs, tracing live calls, asking people what the system was supposed to do and watching them disagree with each other.
So now, before I agree to any rewrite, I ask what the system does that nobody can state out loud. Not the documented behaviour. The behaviour that exists because of a production incident three years ago, or a client who needed one specific thing, or a bug that became load-bearing once something downstream started depending on it.
If the answer is "we're not sure", the first phase is characterisation rather than a rewrite: write tests against the current system until the tests describe reality instead of intent. Slow and unglamorous, and probably the only thing that keeps the rewrite scoped instead of open-ended.
The other questions I ask are about blast radius, about who owns the fallback when it goes wrong, about whether the rewrite solves a problem the business feels or one only the engineers feel, and about what happens if it stops halfway, because it often does.
I wrote more about how I approach inherited systems over on the blog, if that's useful: https://polycratia.com/c/766896f9
If you have done one of these, tell me what your old system turned out to be doing that the team could not explain.
- $ · permalink
The compliance team kept telling me the moderation queue felt slower every month, and nothing in the data explained it.
Same table, same filters, roughly the same count of pending items. One page took about three seconds to come back.
The permission check ran per row. Every item asked the access layer whether this operator could see it, whether they could act on it, whether their issuance tier covered it. A page of items meant a resolution per item, each one walking roles and permissions. So latency tracked page size and the number of roles in the system, not the amount of data. That is why it crept up so quietly: every new role made every list a little slower.
Caching was not the fix. I resolved the operator's effective permission set once per request, then expressed it as a predicate the query itself could apply. Rows the operator couldn't act on never left the database. Same endpoint, around two hundred milliseconds.
What that had been costing: moderators kept shrinking their page size to keep the screen usable, which made the queue look shorter than it was and pushed real review work out of view. A performance problem had turned into a compliance blind spot.
What I do differently now: when an endpoint is slow, I check first whether latency scales with page size. If it does, the work is per row, and no index will rescue it. Authorization is where I find it most often, probably because it looks like a cheap in-memory call right up until it stops being one.
I write up more of these in long form here: https://polycratia.com/c/53e6c9eb
Worth knowing where your authorization actually runs: inside the query, or per row after the rows come back...
- $ · permalink
The agent passed the tests I wrote for it, so overnight I let it run across a whole product category.
By morning it had produced pages that were fluent, confident and quietly wrong.
Dimensions rewritten into other units. A model number tidied up into something cleaner. A size chart paraphrased into prose that read well and told the customer nothing precise.
Nothing failed. No exception, no retry, no alert. Each output looked good on its own, which is exactly how I had tested it: by reading it myself, one at a time.
That was the whole difference between my test runs and production. In testing, each output had a reader. In production the output goes straight into the database and onto a page, and most of it stays unread. Quality judgment doesn't scale, and a plausible sentence never throws.
So I stopped treating model output as text to review and started treating it as a write to validate. The parts that have to survive untouched (identifiers, numeric values, units, measurements) get extracted before the call and asserted after it. If a value that was supposed to be copied comes back changed, the row isn't written: it goes to a queue for a human. The model gets the prose. It doesn't get the facts.
Testing tells you the agent can do the task. Production asks a different question: what happens to the output that stays unread.
I wrote more about how I structure those invariants, and where they belong in the pipeline, here: https://polycratia.com/c/15a8b2ad
If you're running an agent against real data, it's probably worth knowing which wrong field you would actually notice, and which one would just quietly ship...
- $ · permalink
Six weeks into a monolith-to-services rewrite I asked one question that stopped everything: when a payment lands, which service writes the balance?
Four people, three answers.
The inherited system had one database and no ambiguity. Any module could touch any table, and over the years the write paths had braided together: the payment handler adjusted a balance, a reconciliation job adjusted the same balance, an admin action adjusted it again, and the order of those writes was whatever the schedule happened to produce. It worked because a single transaction boundary quietly forgave all of it.
Split that into services and the forgiveness is gone. Two services writing the same number is not a refactor, it is a distributed consistency problem you volunteered for.
What it cost: I had to stop, map every write path against actual production traffic, and then find that one of them existed only to patch a rounding mismatch nobody had documented. That patch was load-bearing. It would have been dropped in the rewrite, and it would have surfaced as reconciliation drift weeks later, far from anything I had touched...
What I do differently now. Before any rewrite I answer five things, and none of them are about the stack.
Which entities carry money or legal state, and which single component owns each write.
What the system does today that nobody can state out loud: the undocumented behaviour that lives in code, not in tests.
Which of the current boundaries exist for a reason, and which exist because someone was in a hurry in 2019.
What the invariant is that must hold during the migration, while the old path and the new one are both live.
How I prove the new path matches the old one on real traffic before anything gets cut over.
Skip the ownership question and what you ship is the same confusion as before, spread across more network hops.
I write up more of these migration post-mortems on my blog, if that's useful: https://polycratia.com/c/20ba6814
If you've done a monolith split with money in the system, what did you run in parallel to prove the new write path was correct?
- $ · permalink
Saying "it works on my machine" is rarely about the machine. You are declaring that your code depends on something it did not ask for.
A runtime hands your process a pile of implicit defaults: a timezone, a locale, a filesystem encoding, a DNS resolver, a certificate bundle, a clock that may or may not be synced, a float formatting rule, a dictionary ordering that happens to be stable this week. None of that sits in your repository. All of it sits in your behaviour.
So the failure mode has nothing to do with your laptop being special. The set of inputs to your program is simply larger than the set of inputs you wrote down.
In payments this gets expensive fast. Amount parsing that leans on the host locale turns a decimal separator into a thousands separator. A timestamp written without an explicit zone lands a transaction in the wrong settlement day, and reconciliation against what the bank actually settled stops matching. A certificate bundle that exists on a developer image and not on a production container turns an outbound payment provider call into a TLS failure that reads like the vendor being down. The code is identical. The inputs were not, and they were not declared anywhere.
Catalog and delivery pipelines get the same class of problem. A parser that worked on one machine was relying on the filesystem encoding of the box it ran on, not on anything in the source document.
How I approach it: treat every environment-dependent value as an explicit input, not a convenience the runtime provides. Pin timezones in code, not in the host. Parse money from a declared format into a decimal type, not through a locale-aware helper. Load trust stores and credentials by explicit path, and fail loudly at startup if they are absent. Validate configuration on boot, so the process refuses to run misconfigured instead of running wrong. And containerise to shrink the undeclared surface rather than to "solve environments": Docker does not make the implicit explicit, it just makes it the same implicit everywhere, which is a different and weaker guarantee.
The idea worth keeping: reproducibility is not a property of the environment, it is a property of how honest your code is about its dependencies. A system that only behaves correctly under defaults it never names is a system that happens to work.
I have written more about this class of production failure on my blog: https://polycratia.com/c/2bc59dd0
If you have chased one of these, I would guess it was the clock, the locale or the trust store...
- $ · permalink
A three-second endpoint is usually not doing three seconds of work. It is waiting for a lock.
It hides well, because everything you instrument looks healthy. CPU flat, query plan fine, per-statement execution time in milliseconds: lock acquisition is not execution, and a lot of tooling reports only the latter. The time disappears between request received and statement started, and that is exactly where most traces carry no span.
What gives it away is the shape of the latency, not the size. Contention is bimodal: almost every request is fast, a minority takes seconds, and the slow ones cluster by clock time rather than by traffic. When your p99 spikes line up with a settlement or reconciliation run instead of with peak load, that is not a performance problem. It is two writers meeting on the same row.
In ledgering work this is the default failure mode. A mutable balance row is a global serialization point: every credit, debit and attribution update competes for it, and one long-running job holding it blocks reads that only wanted a number. The fixes are structural rather than clever. Keep transactions short enough that no business logic or outbound call runs inside them, acquire locks in a consistent order, and prefer append-only entries with derived balances over one hot row everyone has to update.
Slow code you can profile. Contention you have to design away.
I write up more of these production patterns here: https://polycratia.com/c/67624695
Balances, inventory, counters: you probably already know which one surfaces first in your system...
- $ · permalink
A shipping-rates endpoint answered in roughly three seconds, and the working assumption was that the carriers were just slow.
They were slow. Not three seconds slow.
So I put timing around every outbound call, and found two problems stacked on each other. The rate requests went out to the carriers one after another, so the endpoint cost the sum of all the integrations instead of the slowest one. Each request also opened a fresh HTTPS connection, new handshake and all, because the client was getting constructed per call. Next to that, the handler logic was a rounding error.
The fix was unglamorous: persistent sessions per carrier, with the fan-out running concurrently. On top of that a hard time budget across the whole set, so one degraded carrier returns no quote instead of holding checkout hostage.
The real cost was the weeks before. The slowness lived in conversations as a vendor problem, which is a comfortable story, and comfortable stories stop anyone from looking at the part that was mine to fix.
So now, on any endpoint that talks to third parties, outbound calls get their own timing from day one, measured separately from handler time. If I cannot say how many milliseconds went to waiting on someone else's network, I do not know where the latency went. I am guessing with extra steps.
I write up more of these production post-mortems here: https://polycratia.com/c/0a56a981
Your request path probably still waits in sequence somewhere for things it could be asking for at once...
- $ · 9 min readCite the row, or say unknown: a checkable crypto classifier
A cryptography inventory that prints RSA quantum_vulnerable is asking to be believed. Whether you can check it instead comes down to two design decisions that have nothing to do with scanning:…
- $ · permalink
Three seconds on the clock is rarely three seconds of work. It's three seconds of waiting for a resource something else is holding.
The pattern I hit most often in payment and verification endpoints: a database transaction opened at the top of the handler, then an outbound call to a provider made inside that transaction. Nothing is computing. A connection sits pinned while a third party decides how fast it wants to answer.
Give that a modest pool and a provider having a bad minute, and the pool drains. Now endpoints that touch no provider at all start queueing on connection acquisition. In your timings it surfaces as slow SQL or slow views, so you go optimize a query that was never the problem. The slow thing was not the thing doing work.
So before I touch indexes, I check one thing: does any request path hold a transaction across a network boundary I don't control? If yes, that's the three seconds, and no amount of query tuning will move it.
What fixes it is boundary discipline, which isn't performance work at all. Write local state, commit, then call out. Treat the provider's answer as a separate state transition with its own retry path and its own idempotency key. The handler gets fast again because it stops owning someone else's latency.
And measure acquisition wait separately from execution time. A saturated pool and a genuinely slow query look identical in average response time and have nothing in common: one is a concurrency bug, the other is a data-access bug.
I write up more of these production trade-offs on my blog: https://polycratia.com/c/1f535fc1
Somewhere in your request path, a transaction is probably still open across a call you don't control...
- $ · permalink
Three seconds down to 200 milliseconds is almost never the same endpoint going faster. It is a different endpoint.
The slow version answered the question honestly: it asked several carriers for live shipping rates, waited on the slowest one, and returned a fresh aggregate. No single call was broken. The latency was the design, because the response could not exist until every upstream had answered.
The fast version skips that work entirely. Rates get pulled on a schedule, normalized into one internal shape, and stored with a freshness stamp. The request reads a precomputed row. 200ms is a cache read wearing an API's clothes.
What you buy with that is a new obligation.
You now owe the caller a freshness contract: how stale a quote may be, which operations may use a stale one (showing options during checkout), and which must force a live call (buying a label, charging the customer).
You also owe yourself a visible failure mode for when the background job falls behind. Serving yesterday's prices quietly is worse than being slow, because nothing errors.
So the honest version of the claim goes like this. I did not remove three seconds. I moved them off the request path onto a job nobody is waiting on, and I paid for it with staleness I had to make explicit in the data model.
Most latency work is that trade rather than a clever optimization. Treat it as a product decision and the engineering gets simple.
I write up more of these mechanics on my blog: https://polycratia.com/c/bafb9eaf
If you have made this swap, I'd like to know where you drew the line between a cached quote and a live one...
- $ · permalink
The signature was valid the day I made it. Three months later it was invalid, and nothing in the system had changed.
A counterparty's verifier rejected a batch of signed documents. My verifier accepted the same files. Same bytes, two verdicts, which cannot happen, so one of those statements was wrong.
Mine was. The bytes were not the same.
Between the step that produced the document and the step that stored it sat a normalization pass. Not a feature, not a decision: a serializer option left at its default. Signing ran on the in-memory document. Storage happened after normalization. At creation I verified the signature against what I had just signed, so every test passed and every integration check passed. The discrepancy lived only in the artifact I kept.
And nobody verifies a signed document on the day it is signed. They verify when a deal closes, or when someone disputes it. By the time the system tells you it is broken, the broken documents sit in other people's hands already, and the fix is not a deploy: it is going back to real counterparties and asking them to sign again.
What I do differently now: the signature gets computed over the stored artifact, never the in-flight one. A job re-verifies stored documents against stored signatures on a schedule, not once at creation. Any transform sitting between producing and persisting I treat as signature-breaking until proven otherwise.
The bugs that cost the most are quiet. Detection is the part the business process defers, and the gap between the write and the first real read is the actual blast radius.
I write up more of these failure modes here: https://polycratia.com/c/efba9f13
Probably worth asking where your system still assumes that what you validated is what you stored...
- $ · permalink
The endpoint took three seconds and came back 200 OK. The 200 was the part that bothered me.
A KYC flow: submit a document, get a verification decision back. Latency had crept up over a week. Nothing on the error dashboard, no alerts, nothing loud enough to page anyone. Just slow.
The handler was fine. The queries were fine.
What I ended up reading was the client wrapper around the verification provider. Inside it sat a retry decorator added years earlier: three attempts, a sleep between them, and a swallow on anything that looked transient.
The provider had been failing calls and succeeding on retry for days. The service absorbed all of it. The only trace left anywhere was wall clock time.
Two costs. I burned a day tuning the layer that wasn't broken, because I read query plans before I read attempt counts. And the provider's submit endpoint wasn't idempotent, so every slow success had been creating duplicate submissions on their side, which turned into manual cleanup with their compliance people.
Now I log attempt number and per-attempt duration on every outbound call, and I alert on retry rate instead of latency. Retries wrap only the calls I know are safe to repeat. Everything else fails loudly and early.
A retried call is a failure you decided to hide from yourself. Latency is just where it leaks out.
I write up more of these production failure modes here: https://polycratia.com/c/720dea45
Somewhere in your system there is probably a slow path that turns out to be a retry nobody remembers adding...
- $ · permalink
The bugs that scare me in a payment system are not the ones that fail: they are the ones that succeed twice.
A unique constraint on a nullable column is the cleanest example I know. In Postgres, NULL is not equal to NULL, so uniqueness quietly stops applying to exactly the rows where the external reference is missing. The schema passes review. The constraint sits right there in the migration. It just does not cover the case it was written for.
Then a retry arrives from a channel that sent no reference that day, and the ledger takes a second credit for the same obligation. Nothing throws. No alert fires, because every component did what it was told. The balance is wrong and the logs look healthy.
A bigger constraint does not fix that. Refusing to write a row without a usable key does. I keep the dedup key non-nullable and derive it deterministically from normalized requisites when the provider hands me nothing. Anything that cannot be normalized goes to an unmatched queue where a human sees it, instead of landing in the ledger with a NULL that silently opts out of every guarantee.
Severity has little to do with the size of a bug. It has everything to do with whether the system can tell it happened. The cheap bugs raise exceptions. The expensive ones produce plausible writes.
I write up more of these constraint-level failure modes here: https://polycratia.com/c/ad3da1d6
Probably worth checking which nullable column in your schema is load-bearing for uniqueness right now...
- $ · permalink
"It works on my machine" stopped being an environment problem years ago. Containers settled that. What's left is a cardinality problem.
Your machine has exactly one of everything.
One request in flight. One copy of each callback. One well-formed record. One clock.
Production guarantees none of that, and that's where code which passed locally starts producing wrong results instead of errors.
Three places I see it break, all of them invisible on a laptop:
Concurrency. Locally you are the only actor, so a read-then-write on a balance looks correct every time. Put two simultaneous requests through the same code and it credits twice, because nothing in it ever claimed a row before computing on it.
Delivery multiplicity. You fire the provider callback once, by hand. Real rails retry: on timeouts, on a 500 you returned after the write committed, on their own schedule. A handler that isn't safe to run twice is already broken. You just haven't been retried yet.
Data shape. A catalog parser written against one hand-picked product page is a parser tested on a sample size of one. At scale the fields go missing, prices show up in another format, stock values stop being numbers, and the parser quietly returns a shape nobody downstream validates.
So I stopped testing for environment parity and started testing for multiplicity. Replaying the same event twice is a standard test for me now, not an edge case. Two writers at the same row. The ugliest records I have going into the pipeline, not the cleanest.
What I want is code with no opinion about how many times, in what order, or in how many copies it runs. A laptop that matches production doesn't get me there.
I write up more of these production failure modes on my blog, if you want the longer form: https://polycratia.com/c/88a41cd1
Curious which of the three bit you hardest: concurrency, duplicate delivery, or data shape.
- $ · permalink
In one night the pipeline rewrote more of the catalog than I could read in a week. That ratio was the actual bug.
During testing I was the evaluation layer. I ran a batch, skimmed what came back, and called it good: fluent, on-brand, better than the human-written mess it replaced. Nothing in that loop was automated, and nothing in the test setup told me it needed to be.
In production the model stayed exactly as good. What changed: the reading stopped.
And the failures that matter in a product catalog are not bad prose. They are confident edits to the parts that aren't language. A unit quietly normalized. An attribute dropped because it didn't fit the sentence, or a size "clarified" into something the supplier didn't say. Fluent output, wrong record. Those fields feed search facets and carrier rate lookups downstream, so a language mistake stops being a language mistake.
The cost was a re-run over part of the catalog, plus everything that was live in between.
What I do differently now: I gave up on grading quality in production and check invariants instead. Numbers present in the source have to be present in the output. Units unchanged. Attribute keys and enumerated values pass through the pipeline untouched, since they don't go through the model at all. Output length inside a bound derived from the source. Anything that fails lands in a review queue instead of on the storefront.
That is a much smaller ambition than verifying correctness. It is probably also the only part that holds at volume.
I wrote up more on how I structure that layer here: https://polycratia.com/c/202314b6
If you run an LLM step in production, I'd want to hear the one invariant you check on every output.
- $ · permalink
I inherited a Python 2 system, and the migration plan was already written when I got there. Six weeks, module by module, tests green at the end of each one.
What the plan did not account for: the system had been in production long enough to accumulate behaviour nobody had written down.
A rounding convention that existed only because one integration expected it. A date-parsing path that quietly accepted two formats, because an upstream partner changed theirs years earlier and nobody fixed the sender. A background job that leaned on the order rows came back in, since that ordering happened to be stable under the old driver.
None of that lived in the tests. The tests covered the code as the authors understood it. The undocumented behaviour was load-bearing anyway: real integrations had been built on top of it.
So the migration kept passing CI and kept producing small wrong results in production. Not crashes. Numbers that looked plausible and were off.
What I do differently now: before any rewrite, I spend time establishing what the current system actually does, not what it was designed to do. Capture real inputs and real outputs from production, replay them against the new code, and diff. Not assert. Diff. What matters is the differences you did not predict, and you cannot write an assertion for a behaviour you do not know exists.
That changes which questions are worth asking before a rewrite. Not "is the architecture wrong": it usually is, that is why you are here. The useful ones are narrower.
What does this system do that is not in any test or document?
Which of those behaviours does an external integration depend on?
Can I observe current behaviour well enough to compare against?
What is the smallest piece I can replace while both versions run?
And: if the rewrite is wrong, how do I find out, from an alert or from a user?
That last one decides whether a rewrite is survivable. A rewrite that fails loudly costs you a bad week. One that fails quietly against money or documents is something you find out about months later, in data you now have to reconstruct.
Monolith-to-services work has the same shape, just distributed. The seam you cut is the seam where the undocumented coupling lived.
I wrote more about how I approach inherited systems and migration seams on my blog: https://polycratia.com/c/cf819e95
For those of you who have done a large migration: what caught you that was not in any test?
- $ · permalink
An agent that passes testing and then breaks in production usually hasn't gotten worse at reasoning. It has started repeating itself.
In a test run every step executes exactly once. State is clean, there is one pass and one actor. Production takes away all three guarantees. A gateway times out and retries. A worker restarts and the queue redelivers the message. The agent itself re-plans after a partial failure and takes the step again, because from where it sits the step never completed.
Repeating a model call costs you nothing. Repeating a tool call that writes is a different story.
And most agent frameworks treat tool calls as function returns: a result, nothing more. There is no notion of an effect that already landed. So the second execution creates a second record, the agent reads state that contradicts its own plan, and corrects it with a third write. The logs read like a reasoning failure. It is a delivery failure.
I handle this the way I handle payment endpoints. Every writing tool gets an idempotency key derived from the intent of the step, not generated at call time, because a random key is new on every replay and protects nothing. The effect goes into a log keyed by that value, and the tool checks the log before acting, returning the previous result instead of acting twice.
That one discipline is what makes an agent safe to re-run, and safe to re-run is what production is asking for.
Agent readiness is a side-effects problem long before it is a prompt problem.
I keep writing up these production-side notes here: https://polycratia.com/c/f4382bc1
If you run agents with real write access, I'd like to know what you key idempotency on: the step intent, the plan node, or the task id...
- $ · permalink
A fast database and a slow API do not contradict each other. The units just do not match.
Your database dashboard reports latency per statement. Your users feel latency per request. Nothing reconciles those two numbers for you, and the gap between them is where the time actually goes.
A request that fires two hundred queries at 2ms each gives you a green database and a 400ms endpoint. No single operation is slow. Everything is sequential.
The same arithmetic repeats at every other boundary in the request path. Serializing a deep object graph. A fan-out to carrier rate APIs, where your latency is the worst case of the slowest carrier rather than the average across all of them. A cache that is instant on a hit and a full recompute on a miss, so your median and your tail end up describing two different systems.
So I start with the count rather than with query time: how many round trips does one request make, to the database and to anything else reached over a socket? Request latency is dominated by the number of boundaries crossed, not by the cost of crossing one.
Which means the fix is usually shape work rather than tuning. Hydrate in sets instead of per row. Collapse read paths that were convenient to write and expensive to walk. Move fan-out off the response path and let the client ask for the result when it exists. Index work helps, but it only makes a cheap crossing cheaper.
This is also why "the database is fine" is such a common conclusion, and such an expensive one. It is true, and it closes the investigation one layer above the problem.
I write up more of these production latency patterns on my blog, if you want the longer version: https://polycratia.com/c/f8e6a3ba
The last time you chased a slow endpoint, what showed you the real cost was probably request-level tracing, or you sat there counting the queries by hand...
- $ · 9 min readValidate VEX on the way out, not on the way in
A tool that both reads and writes VEX documents ends up with one struct carrying two opposite obligations. A document I issue is a claim somebody downstream will act on, so anything ambiguous has to…
- $ · permalink
Your agent didn't get dumber in production. It just started getting a second attempt at actions you only ever tested once.
In testing, an agent run is a clean line: one prompt, one tool call, one response, one outcome you check by hand. In production that same run is a distributed system. The model times out mid-call and the client retries. A step fails halfway and the orchestrator replays the plan from the last checkpoint. A user hits refresh and starts a duplicate. The reasoning is identical. The side effects are not.
I spent years handling this same failure class in payments, and the lesson carries over directly: the dangerous part of an operation is the write, not the decision. A charge, a payout, a status transition, a sent notification. Retry a read and you waste a little money. Retry a write and you have corrupted state that no amount of better prompting will repair.
So when I build agent pipelines, I stop treating tool calls as function calls and start treating them as money movements.
Every action that mutates anything carries a caller-supplied idempotency key derived from the intent, not from the attempt. A second call with the same key returns the first result instead of doing the work again. Actions get logged before execution, not after, so a crash leaves evidence rather than ambiguity. And anything the agent cannot safely redo (an outbound message, an irreversible transfer) sits behind an explicit confirm step owned by the system, not by the model's judgement.
The uncomfortable consequence: your agent's capability is capped by how many of its tools are replay-safe. Not by the model. I have watched perfectly good pipelines get quietly downgraded to read-only recommendations because nobody wanted to own the write path.
The other half of the gap is that tests give an agent well-formed inputs. Production gives it a truncated OCR result, a catalog field in the wrong language, a provider returning 200 with an empty body. The agent does not crash on these. It reasons over them confidently and produces a plausible wrong action, which is strictly worse than an exception, because an exception gets caught and a plausible wrong action gets executed.
My working rule: an agent is production-ready when every tool it can call is safe to call twice, and every input it can receive is one it will refuse rather than guess at.
I write up more of these production trade-offs here: https://polycratia.com/c/d7220ccc
If you run agents with real write access, I'd want to hear where your first duplicate action came from: a client retry, an orchestrator replay, or a user double-submit...
- $ · permalink
Three seconds down to 200 milliseconds, and almost none of that came from faster code.
The endpoint was a product listing that assembled its own answer on every read: catalog data, availability, and live delivery rates fanned out to several carriers. Each piece was fine on its own. Serially, inside one request, they were not.
So I moved the work instead of optimizing it.
Carrier rates turned into precomputed quotes, refreshed on their own schedule and keyed by the few dimensions that actually move a price. Derived fields that used to be recalculated per request got materialized at write time. The response shrank to what the caller renders, not what the ORM was happy to serialize. What was left in the request path was a lookup.
The part nobody puts in the before/after screenshot: you just bought staleness. A precomputed rate can be wrong by the time you see it, and a materialized field can drift from its source. That has to be designed and not discovered: explicit freshness windows, a path that flags a stale or missing quote instead of quietly serving it, and invalidation that rides along with the write rather than a cron job hoping to catch up.
200ms is not a performance achievement. It is what follows from deciding what a request is allowed to do.
I went deeper on where this trade-off breaks over here: https://polycratia.com/c/7f471073
I'd be curious where you draw your own line between computing on read and materializing on write...
- $ · permalink
The same signature verified fine on my laptop and came back invalid on the server. Same document, same key, same code path.
This was an e-signature service built to national digital-signature standards, with the document workflow in Go. The verification step did something that looked fine in review: it rebuilt the PDF from the template plus the stored form data, hashed the result, and compared that hash against the one sealed in the signature.
Locally the rebuilt bytes matched every time. On the server they didn't. The host had a different font available, the renderer fell back, one line of a table wrapped a millimetre differently, and the byte stream changed. The hash changed with it. So a document nobody had touched since signing was now reported as tampered.
The fix was cheap. The expensive part was the two days I spent staring at keys, clock skew and encoding, because the error said "invalid signature" and an invalid signature is supposed to mean something serious. And what I then had to go explain was worse than a bug: for a while, whether a legally binding document was valid depended on which machine asked.
What I changed: the signed artifact is bytes, not a recipe. At signing time the rendered document gets stored once, immutably, and that stored object is the only thing verification ever reads. I treat generation as non-deterministic by default, because it is: fonts, library versions, locale, image codecs, timestamps all leak in.
The general shape of it: "works on my machine" usually means the machine is an input you never declared. In document pipelines that input gets baked into something you later have to prove in front of someone.
So the question I now ask early on any pipeline that produces an artifact of record: can this artifact be produced twice? And if yes, which copy is the truth?
I write up more of these document and payment pipeline failures here: https://polycratia.com/c/3367aff6
Somewhere in your system something still gets regenerated when it should have been stored...
- $ · permalink
The endpoint ran 1 database query. It still took three seconds.
That's exactly why it hid for months. I was watching query counts, and the count was spotless: one lookup, one object out. No slow query log, no missing index, nothing to blame.
The cost sat in the serializer. This was a deal listing on a high-ticket asset marketplace, and every listing carried a small completeness badge (whether the deal's documents were signed and current). To compute that badge, the serializer asked the document service about each document in turn. Twenty-something calls per listing, each one fast, all of them over the network, all of them sequential. Nobody wrote a loop that looked like a loop. It was a property on a nested field.
Two changes took it to about 200ms: one batched call in place of the per-document calls, and a narrower response contract, because the client rendering the list never needed the full document set. It needed a flag.
What I do differently now: I instrument outbound calls per request the same way I instrument database queries. An N+1 inside a process is noisy and easy to catch. An N+1 across a service boundary is polite and well-typed, and invisible to every counter you already trust.
I write up more of these in long form on my blog: https://polycratia.com/c/ec39f919
Where your request-level outbound calls actually get counted, if they get counted at all...
- $ · permalink
The database graphs were boring: fast queries, nothing in the slow query log, no lock waits. And the moderation panel the compliance team uses was still crawling, sometimes it just hung.
So I went looking somewhere else and found the app was out of connections. Every worker held one. Not to run a query: to wait.
The code did something that reads perfectly well: open a transaction, write the verification record, call the KYC provider over HTTP, write the result, commit. One atomic unit, no half-finished checks. I had written that shape myself more than once.
What it actually does is pin a database connection for the full duration of someone else's network latency. The provider got slow and my pool drained. Every unrelated request queued behind it, including the cheap reads that needed the database for a millisecond. The database sat idle while the API was down.
What I do differently now: transaction boundaries don't contain a network call. Write intent, commit, call out, then open a second short transaction to record the outcome, and make the in-between state an explicit step in the state machine instead of an accident. I graph pool wait time next to query time, because they fail in opposite directions.
An API's latency is a sum of waits, and the database dashboard reports exactly one of them. The expensive ones are usually for resources the database owns but your code is holding.
I write up more of these production post-mortems on my blog: https://polycratia.com/c/64efe93c
Your pool wait shows up somewhere today: in a metric, or only in user complaints...
- $ · permalink
The auto-dialer rang a borrower who had already paid that morning.
Not a one-off. The whole morning batch ran off a list that went stale before the first call even connected.
The bug was one line. The collections job queried overdue loans at the start of the run, held that list, and worked down it. Payments landing mid-run never invalidated anything, and bank attribution lags behind the money anyway. So someone who settled at opening got a warning notification, and then a call.
Nothing crashed. No alert fired. The job completed and the queue drained, and delivery rates looked normal. The system did exactly what it was written to do.
The cost was not technical. You can roll back a deploy. You cannot un-call someone and explain that you were wrong about their debt. That is trust spent on exactly the users you need to keep paying, and in collections it is a compliance surface, not just an apology.
What I do differently now: anything that reaches a human gets re-checked against authoritative state at dispatch time, not at list-build time. The eligibility check and the "sent" record live in one transaction, keyed by user plus action plus day, so a retry can't double-notify. And an outbound channel gets a suppression list and a rate cap before it gets features.
Batch jobs quietly assume the world holds still while they run. Anything with a side effect outside the database should probably assume the opposite.
I write these up in longer form here: https://polycratia.com/c/04a5e36e
If you run dunning or collections flows, the thing worth checking is where your final eligibility check sits: at list build, or at send...
- $ · permalink
An API that answers in three seconds is usually not doing three seconds of work. It is waiting on its own retry policy.
The mechanics are dull. A handler makes one outbound call: a verification provider, a card processor, a bank endpoint. That call has a timeout, and around it sits a retry with a couple of attempts. On the happy path you notice nothing. When the dependency degrades, every attempt pays the full timeout, and the handler quietly becomes the sum of them. The slow response is the system doing what it was configured to do.
What makes this hard to see is where the time lives. The dependency's tail becomes your median. Your own signals look fine: CPU flat, no slow queries, handler code untouched since last quarter. The latency is spent waiting, and waiting does not show up in the places engineers habitually look.
Two changes have fixed this for me in payment and KYC integrations.
A deadline instead of timeouts. The request gets one budget at the edge, and every downstream call inherits whatever is left of it. An attempt that cannot finish inside the remaining budget does not get made at all.
Retries off the request path. With idempotency keys on the outbound call, the second attempt belongs to a worker and not to the user's open connection. The caller gets a determinate answer now, and the system keeps working behind it.
The general shape: a retry inside a synchronous handler turns an availability feature into a latency bug, and it does that silently.
I write up more of how I design these request paths on my blog: https://polycratia.com/c/4c6f9622
I'd go find out who owns your retry budget: the client, the handler, or nobody in particular...
- $ · permalink
A three-second response is rarely three seconds of work. Usually it is a few milliseconds of work, waiting its turn.
The signature is easy to read once you know it: the endpoint's query time looks fine in isolation, CPU is unremarkable, and latency climbs with concurrency instead of with payload size. That is not a slow system. It is a serialized one.
In ledgering code the serialization point is almost always one hot row. Every request that touches an account's balance takes a row lock, and while holding it the handler does other work too: writes an audit entry, fans out a notification, computes a pro-rata split across multiple investors in a deal. Each step is fast by itself. Together they stretch the critical section, and under load every arriving request pays for all the ones queued ahead of it.
Two changes matter, in this order.
First, move everything out of the critical section that does not have to be in there, starting with anything outbound. A network call inside a lock turns someone else's latency into your throughput ceiling.
Second, stop locking a mutable total. Append immutable entries and derive the balance, and concurrent writers stop competing for the same row at all.
The habit worth keeping: measure lock wait as its own number, separate from query duration. Most dashboards report how long statements ran, not how long requests sat before they were allowed to run, which is exactly where the three seconds live.
I write more about how I design attribution and ledgering here: https://polycratia.com/c/f7588593
So it is worth knowing where your write path serializes first: one hot row, one queue, or one external call inside a transaction...
- $ · permalink
A repayment came in, got split across the investors in the deal, and the ledger came out one cent short. Not every time. Maybe one deal in forty.
This was a P2P lending system: a borrower repays, and the payment goes out pro-rata to everyone who funded that loan. The math looks trivial. Each investor gets their share of the principal and interest, proportional to what they put in.
The bug was that each share got rounded on its own, independently, and then written. Four investors, four roundings, and the sum of the parts no longer equalled the amount that actually arrived. Sometimes a cent sat unassigned. Sometimes a cent got invented.
Nothing threw. No alert, no failed transaction. Investors saw plausible numbers. It only ever surfaced in reconciliation against what the bank actually settled, and by then it had been happening quietly for a while across many deals.
The money did not matter. A cent is nothing. What it costs you is this: once you find an unexplained cent, you can no longer tell anyone which balances are right. Every number in the system becomes a maybe until you prove otherwise, and proving otherwise means replaying history.
What I do differently now: money is integer minor units, never floats. Shares are computed as integers, and the remainder is allocated deterministically by a fixed rule rather than dropped by rounding. And the invariant (the parts sum exactly to the whole) is asserted inside the same transaction that writes them. If it fails, nothing is written and the distribution is flagged for a human.
The general lesson I keep relearning: a conservation rule that only holds on average is not a rule. In a ledger it has to hold on every single write, or you are just deferring the discovery.
I write up more of these production post-mortems on my blog, if that is your kind of reading: https://polycratia.com/c/080b3ee0
If you run ledgers, I am curious where you enforce your conservation invariants: in the database as a constraint, in the application transaction, or in a nightly job that tells you after the fact.
- $ · permalink
A Python 2 monolith landed on me, and everyone agreed it had to be rewritten. I agreed too. That agreement was the easiest part of the whole project.
I mapped the modules, drew the service boundaries, picked the order. The plan looked clean. Then the first extracted service went live and started producing results that were technically correct and operationally wrong.
The old code had a behavior nobody had written down: an upstream call came back empty, and it fell through to a stale cached value instead of failing. Someone did that years earlier, probably for a good reason, under pressure. Downstream, three different consumers had quietly built on top of it. One of them treated a missing record as a signal, not as an error.
My rewrite was more correct than the original. That was the problem. It raised where the old one shrugged, and the shrug had become part of the contract.
What it cost: weeks of chasing symptoms that looked like data issues and were really semantic drift between two versions of the same system.
What I do differently now, before touching anything:
I stop asking whether the code is bad. It is, that's why I'm here. I start asking which of its current behaviors are load-bearing, and I don't look for the answer in the source. The source tells you what someone intended. It does not tell you what consumers came to depend on.
So I run both systems against the same real traffic and diff the outputs. Not the happy path: the empty results, the timeouts, the duplicates, the partial writes. Every place the two disagree is either a bug I'm fixing or a contract I'm about to break, and I have to decide which, explicitly, in writing. The ones I choose to break get a migration instead of a surprise.
That shadow-run period feels like pure overhead when you're trying to ship. It is the cheapest part of the entire rewrite.
The second thing I ask: can this be strangled incrementally, or does it have to go in one cut? If the answer is one cut, the honest follow-up is whether the business can absorb a bad week. Sometimes it can. More often nobody asked, and finding out during the cutover is a career-shaped event.
I wrote up the full set of questions I run through before agreeing to a rewrite, including the two that usually kill the idea, here: https://polycratia.com/c/880960cd
For those of you who've done a big migration: I'd like to hear which undocumented behavior turned out to be a contract you didn't know you'd signed...
- $ · 9 min readRouting a withdrawal is a value, not an if inside the sender
A withdrawal to an address you already custody does not have to touch a chain: crediting it is a ledger move, with no fee, no confirmations and nothing to wait for. A withdrawal to anyone else is a…
- $ · permalink
"Works on my machine" is usually a claim about concurrency, not about dependencies.
Locally you run one process and one worker, a single request at a time. Every race in the code resolves correctly by accident, because there is nothing to race against. Production runs that same code behind a worker pool, and the accident stops happening.
The mechanics are almost always check-then-write. Your handler reads whether a payment was already attributed, finds nothing, then writes the attribution. On a single process the read and the write are effectively atomic. On four workers pulling off the same queue, both can pass the check before either writes, and the ledger ends up with the same money attributed twice.
Nothing throws. No stack trace. It shows up days later as a reconciliation mismatch, which is the expensive way to learn about it.
Retries sharpen it. A broker redelivery means the duplicate does not need two users racing: one user and a dropped ack is enough.
So I stop enforcing uniqueness in application code. A unique constraint on the idempotency key in the database, insert first, and treat the conflict as "already done" instead of reading first and then deciding. The database is the only participant that sees every worker.
And I run it locally the way it runs in production: worker count turned up, the same message delivered twice on purpose. A green test suite at concurrency one is not evidence of correctness. It is an environment too small to expose the race...
I unpack more of these quiet failure modes on my blog: https://polycratia.com/c/51fdc28e
So where do you draw the line between the invariants your database enforces and the invariants your service layer owns?
- $ · permalink
The reconciliation job ran clean on my laptop for a week. Staging matched every payment. Then it ran one night in production and reported settled payments with nothing to attribute them to.
The matching logic was fine.
My laptop was on local time. The container was on UTC. The job cut the day at midnight, and midnight was a different instant in each place. Payments landing in the last hours of the local day fell into the next UTC window, so they got compared against a settlement file that hadn't seen them yet.
That unmatched set isn't a bug report. It's a question somebody has to answer by hand: did money arrive that I can't account for? It costs someone an afternoon, and it costs everyone a little trust in the numbers, which is the expensive part. Trust in reconciliation output is the whole reason the job exists.
What I do differently now.
No time arithmetic reads the host's zone anymore.
Day boundaries are an explicit property of the rail: a configured cutoff and zone per source, stored alongside the window, so a report can tell you which boundary produced it.
And the test suite runs with the timezone set to something deliberately wrong. If results change, the code is reading something it shouldn't.
Environment differences rarely announce themselves. They don't crash: they shift windows, round differently, order rows differently. The output still looks plausible, and that's why it reaches someone's eyes before it reaches your logs.
I write up more of these production-first patterns on my blog: https://polycratia.com/c/9178de1a
Host-level defaults bite quietly, and timezone is only the one that got me: locale and collation are sitting right there too...
- $ · permalink
The parser ran clean on my laptop for a week. Deployed, it kept running clean, and quietly wrote prices into the catalog that were wrong by a factor I didn't catch for two days.
Nothing crashed. The selectors matched. The numbers were numbers.
What changed was who the request looked like. From my machine the retailer served one page variant. From the production egress it served another: different markup in places, and a price field formatted for a different locale, with the currency sitting in an element I never parsed because on my machine it wasn't there.
So the pipeline ingested plausible values. Plausible is the dangerous category: a malformed price throws, and a plausible one ships to customers and into every downstream margin calculation. The hour it took to fix was the cheap part. Reprocessing a catalog cost more, and so did not knowing, for a while, how far the bad values had travelled.
What I do differently now on anything that scrapes: the parser asserts on the contract, not just the selector. Currency explicit or reject. Expected ranges per category. Plus a small canary set of known items that the pipeline rechecks on every run, and if those drift the run stops instead of writing.
The local environment lies most when it sits on a friendlier network than production.
I've written up more of these failure modes from payments and catalog work on my blog, if that's useful: https://polycratia.com/c/8d5d683b
If you run scrapers at scale, you either validate parsed output against a known baseline or you hear about it from the people downstream.
- $ · permalink
The hosted payment page passed every test I ran on my laptop. Same card, same flow, green all the way through.
Then it went live under a merchant's own domain and branding, and a slice of payments started finishing nowhere.
Money moved on the rail. The order stayed pending.
What my machine had and production didn't was a single origin. Locally the page, the API and the session all sat on one host, so the cookie carrying the payer's session survived every redirect. In production the page lived on the merchant's domain, the flow left for the bank's step and came back, and on that cross-site return the session was gone. The page rendered a neutral state. The payment was already authorised.
Nothing errored. It surfaced later, in reconciliation: settled money with no obligation attached to it.
Two changes stuck with me after that.
First: local dev doesn't run on localhost alone for anything with a redirect in it. The page gets a real second domain, so cookie context is part of the test instead of an accident of my setup.
Second: the browser's return is a hint, not a fact. Payment state comes from the rail, reconciled server-side, and the page just reflects what reconciliation says.
I write up more of these production-versus-local gaps on my blog: https://polycratia.com/c/c6e330e9
If you ship checkout flows you probably have your own version of this, whether it was a redirect, a timezone or a worker count. Mine was the redirect.
- $ · permalink
My onramp passed every test I had, because the verification provider's sandbox answered instantly, and it answered yes almost every time.
In code I had two paths: approved and rejected. Both covered. Both worked.
Then real users showed up, and the compliance queue filled with people who were neither.
Manual review was the state I had barely modelled. A document that needs a human. A callback that lands hours later instead of seconds. A re-request for a clearer photo of the document the user already uploaded. Sometimes a verdict that flips after a moderator takes a second look.
Nothing crashed, and that was the problem. The fiat leg sat waiting on a decision that was not coming, the interface had no honest way to say "we're still looking at this", and support had nothing to tell anyone.
What I do differently now: I map the provider's full state machine before I write the happy path, and pending becomes the default branch instead of the exception. I store every raw callback, so I can replay real payloads against new code rather than trust a sandbox's optimism. Moderation tooling ships in the same sprint as the integration, because a compliance team with no view into the intermediate states is an outage with a human face on it.
Sandboxes don't simulate judgement. They simulate throughput.
I write up more of these integration realities on my blog, if you find that useful: https://polycratia.com/c/3877661a
If you're running KYC in production, you probably already know the incident that made you take the pending state seriously.
- $ · permalink
Most of the time "works on my machine" is a statement about the age of your data, not about your dependencies.
Your local database was built recently, by the current migrations. Every row in it was written by the code you are running right now, so every row has exactly the shape that code expects.
Production holds rows written by versions of the system that are gone.
A column added as nullable three deploys ago and backfilled "later". A status field that picked up a new state after half the records were already written. A ledger entry from before a reference field became mandatory. Current code reads those rows and branches into a path no fixture ever reaches.
Usually nothing crashes. It just quietly takes the wrong branch, and in payments that surfaces much later as an amount nobody can attribute to an obligation, or a reconciliation run where the unmatched bucket grows for reasons the code cannot explain.
So schema history I treat as part of the contract, not as cleanup.
Backfill in the same migration that adds the column, or make the empty value a real named state with explicit handling. Test against a restored, anonymized snapshot instead of hand-built fixtures. In legacy modernization (Python 2 to 3, monolith to services) I assume the oldest rows define the actual contract, not the model definition.
Fixtures encode what the code expects. Production encodes what the code used to do.
I write more about this kind of production reality on my blog: https://polycratia.com/c/c186dc66
The last time old data burned you it was probably a missing backfill, or a state that arrived after the fact...
- $ · 9 min readA paywall is a table of what is sold
Any HTTP payment gate has to answer one question: what happens to a request it has no price for. Most middleware answers that by accident rather than by design. Everything behind the gate is paid,…
- $ · permalink
Your database is fast. Your request is not, because it asks the fast thing hundreds of times.
Per-query metrics are the best alibi in backend engineering. Every statement comes back in milliseconds, the slow query log is empty, the dashboard is green, and the endpoint still takes a second. Nobody can point at the line that costs you that.
The mismatch is in the unit. The database measures queries. The user experiences a request. In between sits a multiplier that no database dashboard tracks: round trips per request.
You see it most in catalog and marketplace code. A list endpoint loads products in one clean query, then resolves a delivery rate per item, then a permission check per item, then a translation or price lookup per item. Each call is honest and fast. Together they are the latency.
Serialization hides the same shape. Nested objects get assembled lazily, so the query count becomes a property of the response tree and not of the handler you can actually read.
What works for me: log round trips per request next to duration, and give every endpoint a budget. A handler that quietly goes from a handful of queries to dozens should fail review the way a slow query would. Fast per unit is not fast per request.
I write more about this kind of production latency work here: https://polycratia.com/c/e398ba92
Worth finding where your own query count quietly explodes: serializers, permission checks, per-item enrichment...
- $ · permalink
Your API can be slow while every query runs in single-digit milliseconds. The time is not in your code at all. It sits in everyone else's.
Most endpoints in payments, compliance and commerce are aggregators. A checkout call fans out to a payment provider. An onboarding call waits on a verification provider. A shipping call asks several carriers for rates. Each one is a network hop into infrastructure you do not own and cannot profile.
The mechanics are unforgiving. Call three partners in sequence from one handler and its tail latency is not the average of theirs: it is the sum of their worst cases. One provider sliding from 200ms to four seconds moves your p99 while your database dashboard stays flat and green. Put a retry policy on top of that and you have doubled the exposure instead of hiding it.
What I do about it: every outbound call gets an explicit time budget rather than a default timeout, and I decide up front which calls are allowed to block a response at all. The rest goes behind a queue or a cached last-known-good value. Carrier rates and verification results tolerate staleness far better than users tolerate a spinner.
Then instrument the gap. Total request time minus your own work is time spent waiting on third parties, and that one number tells you whether optimizing SQL will change anything.
I write up more of these production trade-offs here: https://polycratia.com/c/8bb3532d
If you run aggregating endpoints: do you degrade to a partial response when a provider slows down, or hold the request and hope it answers?
- $ · permalink
The complaints were about an endpoint that did almost nothing: read a document's state, return it.
The query behind it ran in well under a millisecond. No loops, no third-party call, no serialization of anything large. And a few times an hour it took seconds.
I spent longer than I want to admit inside that handler. Added timing around the query. Added timing around the response build. Everything I measured came back fast, and the request was still slow.
The cause was a different route on the same service. It verified signatures on signed documents, a CPU-bound cryptographic check written as an ordinary synchronous call inside an async handler. While that check ran, the event loop did not advance. Nothing else on that worker progressed. Every unrelated request sitting there kept piling up latency that belonged entirely to someone else's code path.
That's what made it so expensive to find. The slow endpoint wasn't slow. It was a bystander. And because the delay got spread across everything in flight, it looked like infrastructure noise rather than one bad call.
Two things changed in how I build async services after that.
First, any CPU-bound work or sync-only client library goes to a thread pool or a queue worker by default, not when it starts hurting, from the first commit. Verification, PDF work, image handling, old SDKs with no async support: all of it leaves the loop.
Second, I instrument event-loop lag as a first-class metric, right next to request duration. If loop lag spikes, no amount of query tuning will help, and knowing that immediately saves days.
The mental shift I found useful: in an async service, latency isn't a property of an endpoint. It's a shared resource, and one blocking call spends everyone's share.
I write up more production failures like this one on my blog: https://polycratia.com/c/d437c2c1
If you run async services in production, are you measuring loop lag, or only request duration?
- $ · permalink
Peak traffic made a category page on a cross-border catalog time out, and the database dashboard stayed flat and green the whole time. Every query in the slow log was fast.
The cache was the problem: specifically, the moment it wasn't there.
Those payloads were expensive to build: catalog rows, carrier rate lookups, currency conversion, serialization. All of it cached, so in steady state the database barely saw the traffic. That is why it looked fast.
The keys had all been warmed by the same job, though, so they expired as a cohort. The instant a popular one went cold, every concurrent request for that category missed, and all of them started the same rebuild. Hundreds of identical computations racing each other, each individually fast, collectively enough to exhaust workers and connections. Endpoints with nothing to do with the catalog queued behind them.
Per-statement timing measures work. A stampede is duplicated work. Nothing in query metrics describes it.
What I do differently now: I jitter every TTL so a warm set doesn't expire together, I put a single-flight lock around recomputation so exactly one request rebuilds while the rest wait on its result, and I serve the stale value while the refresh runs behind it. Stale by a few seconds beats a timeout almost every time.
The uncomfortable part is that a cache makes a system look faster than it is. You aren't measuring your database under load, you are measuring it with most of the load removed. Capacity you never tested is capacity you don't have.
I write up more of these, the failures where the metric was fine and the user wasn't, here: https://polycratia.com/c/39b1693a
If you run read-heavy catalogs, you either guard against the simultaneous miss or you probably just haven't hit the moment where it matters yet...
- $ · permalink
A fast database and a slow API do not contradict each other. Query time measures work, API latency measures waiting.
Queries come back in single-digit milliseconds and the request still takes a second. The time is not in the query. It sits in occupancy: how long each request holds a scarce resource while not actively using it.
The usual shape: a transaction opens early in the handler, and then non-database work happens inside it. A file write to object storage. A call to a verification provider. A document sent for signing. Serializing a large response before the commit.
Every query in that span is still fast. The connection is checked out for the whole span anyway.
So your effective concurrency is not the pool size. It is the pool size divided by how long a connection sits held but idle. Past that point nothing looks broken: the database keeps reporting healthy query times, because requests now queue for a connection before they ever reach a query. That wait shows up in no slow-query log you own.
How I approach it: the transaction is the shortest span in the handler. External I/O and file writes live outside it. Reads that can happen before the write boundary happen before it. In payment flows I think this stops being a style preference, since a hold that keeps a connection open across a provider call converts a partner's bad minute into your outage.
The part worth keeping: stop optimizing query time and start optimizing connection occupancy time. Two different numbers, and only one of them is visible by default.
I unpacked the rest of these patterns here (transaction boundaries, pool sizing, where the waiting actually hides): https://polycratia.com/c/18419e71
If you are profiling an API right now, check whether you measure time spent waiting for a connection separately from time spent executing the query.
- $ · permalink
A catalog endpoint kept timing out on mobile, and the slow-query log for that window was empty.
Postgres was fine: the query came back fast, the plan was clean, the indexes were used. Whatever time I was losing was going somewhere after the rows were already in memory.
So I split the handler's wall time into segments: fetch, object construction, serialization, bytes written. The query turned out to be a rounding error. Most of the budget went into nested serializers expanding related objects row by row, formatting prices and delivery estimates per item, and then shipping a response big enough that a phone on a mobile connection spent longer downloading it than the server spent producing it.
No database metric was going to tell me that. The database measures what it was asked to do. It cannot measure what my code does with the answer, or what the client pays to receive it.
What changed afterwards: the response contract came first. Flat payloads for list views, heavy fields only on the detail endpoint, serialization treated as a cost with a budget instead of a free translation layer. Response size became a reviewable number, the same way query count is.
Now I treat an endpoint as guilty until its time is accounted for end to end, and not just up to the point where the database stops being involved.
I wrote this one up in more detail here: https://polycratia.com/c/2b24e4ff
If you want to know where your slowest endpoint actually spends its time, you need the whole picture: fetch, object graph, bytes on the wire.
- $ · 10 min readThe ledger core takes no lock, and the protocol says whose job it is
Placing a hold is a read followed by a write: you ask what is available on an account, refuse if the amount does not fit, store the hold. Two callers doing that at the same moment can both read the…
- $ · permalink
The payment attribution service I inherited was mostly special cases, and in the rewrite I deleted most of them.
The unmatched pile got bigger, not smaller.
Each branch looked like someone's panic fix. One truncated a reference field before comparing it. One ignored the payer name entirely when amount and timestamp lined up. One treated two same-day transfers as a single obligation. No comments, no tests, nothing written down anywhere.
That wasn't sloppiness: it was production behavior recorded the only way anyone had time to record it. Banks truncate fields. Payers retype references by hand, and a finance team will send one transfer covering two obligations. Every ugly conditional was a bug report from production that I never read.
The cost wasn't an outage, which is what made it slow to admit. A manual reconciliation queue kept growing, people matched payments by eye, and I kept insisting the new code was correct while the balances disagreed.
So before I touch inherited money code now, I stop asking what does this do, and I ask which of these conditions is a requirement I can't see. Every branch gets either a named real-world cause or a historical payment replayed through it to prove it's dead. If it has neither, it stays until it does.
Matching the money is the deliverable. Tidy code is not.
I write these up in longer form here: https://polycratia.com/c/8bee8ffe
I'd bet you have an ugly conditional sitting in your codebase that turned out to be load-bearing.
- $ · permalink
The new service answered every request identically to the old one. I diffed them for weeks.
Then a scheduled job ran twice overnight.
What I was replacing was an inherited monolith with a payment layer bolted on over years. I did the careful version of the migration: mapped the HTTP surface, replayed real traffic against both systems, compared responses field by field. The API was the part I could see, so the API was the part I verified.
What I could not see was the half of the system that no request touches. Periodic tasks. Retry policies. A job that assumed the nightly import had already finished, because on the old box it had, run after run. In the new deployment the workers came up in a different order, and that assumption quietly stopped holding.
Nothing crashed. A notification fan-out fired twice, and state had to be walked back by hand. That is the expensive kind of repair: you are not reading an error, you are reconstructing what the system meant to do.
So my first question before a rewrite is no longer about the code. It is this: what runs here while I am not watching? Each periodic task, its cadence, its retry behavior, and whether it is safe to run twice. A non-idempotent scheduled job isn't a job. It's a landmine with a timer.
The other four questions I ask are ones I only earned by getting this one wrong, and I wrote them up on my blog: https://polycratia.com/c/b4672e53
Somewhere in your system there is probably a scheduled job that hasn't been read in two years...
- $ · permalink
Partway through a monolith-to-services rewrite, the new service started producing record states that its own code could not produce.
I read the new code three times looking for the bug. It was not there.
The tables had other writers. A scheduled job that patched rows nightly. An old admin screen that wrote status fields directly, bypassing every rule the application layer enforced. A runbook with manual SQL in it, which someone ran when support escalated.
None of that lived in the module I was replacing. All of it wrote to the data the module owned.
The expensive part was trust rather than code: once states show up that the new service cannot produce, you can no longer tell a migration defect from a side-channel write, and you reconcile by hand until you can.
So now I inventory writers instead of endpoints, before I touch a line of a rewrite. Every cron. Every admin action. Every integration holding database credentials. Every manual procedure that exists because the product never shipped the button for it.
If a table has writers outside the scope being rewritten, that table is not ready to move. Either the side channels get absorbed into the new service first, or the rewrite inherits them as undocumented behavior and finds them in production.
Legacy systems are hard, I think, for a reason that has little to do with the age of the code: the real interface is wider than the code suggests.
I write up more of these migration lessons on my blog: https://polycratia.com/c/648f7152
When you last moved a module out of a monolith, what turned out to be writing to its data that nobody had documented?
- $ · permalink
Most rewrite checklists ask what your code does. The question that decides the cutover is which of its effects you cannot take back.
Inside your boundary everything is replayable. Re-run a job, re-import a table, diff two responses against the same request.
Outside it, nothing is. A capture on a card rail. A settlement instruction. A KYC check consumed at a provider. A carrier label bought. An on-chain transfer. A signed document. Each one becomes a fact in someone else's system the moment it leaves yours.
That asymmetry is what turns a clean rewrite into an incident. Running both systems in parallel is the safest cutover pattern there is, right up until both of them send. A webhook the old service already acknowledged is a webhook the new one does not see, and the payment sits unattributed. A notification goes out twice because the dedupe key lived in the code you replaced.
So map the boundary before you map the modules: every outbound call and every consumer-acknowledged event, which of them are idempotent on the provider's side and not just on yours, what the natural key is, and which ones have no reversal at all, only a compensating action that can fail on its own.
Then decide which system owns sending during the overlap. One of them. Not both.
Internal behavior you verify by diffing. External effects you verify only by not having duplicated them.
More on the payments and compliance side of this here: https://polycratia.com/c/b1a40859
If you have cut over a system with live money moving through it, you probably already know what owned the outbound side while both were running...
- $ · permalink
Checklists for a rewrite usually ask what the code does. The question that actually breaks rewrites is a different one: which of your identifiers already live in systems you don't control.
Payment references sit in bank statements. Order numbers sit in carrier labels, and in customer inboxes. Document IDs sit inside signed artifacts, where the signature covers the ID itself. Wallet addresses sit in other people's address books, and deposits keep arriving at them long after you stop issuing them.
None of that is in your repository. All of it is a contract.
A rewrite that reissues keys, changes their format or restarts a sequence is not a refactor. It is a silent break in attribution. Money arrives with a reference the new system cannot match. A settlement period stops reconciling. A signed document can no longer be tied to the deal it closed.
The second half of the same question is history. Behavior parity gets you today's requests. Reconciliation, disputes and compliance ask about last year's. If the new system cannot reproduce a past state (the amount as it was attributed then, the status as it stood at the moment of signing), you have replaced an auditable system with one that merely works.
So before you settle the stack, the timeline or the test coverage: list every identifier that has left your boundary, name who holds it, and decide explicitly whether the new system keeps it unchanged or carries a mapping forever. Mapping tables are boring, and they are probably what makes a cutover survivable.
I write more about this kind of trade-off here: https://polycratia.com/c/c19e8ded
Some of your identifiers you would only discover were load-bearing after the cutover...
- $ · permalink
A rewrite is a migration problem wearing a coding problem's clothes.
The questions people ask first (new stack, timeline, test coverage, feature freeze) all assume the work sits in producing the new system. The work sits somewhere else: proving the new system behaves like the old one on inputs that were left undocumented.
So I put this one ahead of those: can both systems process the same request, and can you diff the results?
If yes, the rewrite gets incremental and boring. Route a slice of real traffic to the new path, keep the old one authoritative, compare outputs until each difference is either explained or fixed. Each discrepancy is a free bug report from production.
If no, you are committing to a flag day. And flag days fail on the undocumented parts of the legacy. In Python 2 to 3 migrations and monolith-to-services rewrites, the surprises were rarely syntax or service boundaries. They were implicit behaviors encoded in data: rows that violate the constraint the model claims to enforce, encodings that only show up in records written years earlier, branches kept alive by a single caller.
Three follow-ups matter more than the stack choice. What is the rollback unit: one request, one user, or the whole service? Who owns writes during the overlap, and can anything double-write? What reconciliation tells you the two systems agreed, rather than that neither crashed?
I wrote up how I sequence this kind of migration here: https://polycratia.com/c/f086e021
If you have run a parallel-run migration, I want to hear what the diff caught that your test suite never did...
- $ · 8 min readA post-quantum migration plan should print its arithmetic
A cryptographic inventory lists what you have. A migration plan claims what to do first, and that claim is an opinion wearing a number. If you cannot recompute the score sitting next to a line, you…
- $ · permalink
Your agent didn't get worse in production. It just stopped being the only thing touching the state it was reasoning about.
In testing it reads a record, plans, acts. Nothing else moves. The world it saw at step one is still true at step four.
Production deletes that assumption. Between the read and the write, a user edits the order, a retry fires, a scheduled job closes the window, an operator cancels. The plan stays perfectly coherent. It is just coherent about a world that has expired.
That is not a model quality problem. It is the oldest write-path problem there is, and payment systems had to solve it long before agents existed: never let an actor apply a decision based on state it no longer holds.
What I carry over from payments and deal-lifecycle work:
An agent action is a state transition with a guard, not a function call. The transition declares the state it expects, and it fails loudly once the record has moved.
Every action is idempotent and keyed, so a retried step cannot double-apply.
Re-read at write time instead of plan time, and treat a changed record as a new decision rather than an error to retry past.
Whatever you cannot guard goes to a human queue instead of a guess.
Fixtures are the reason this stays invisible until launch: removing concurrency is most of what makes a test deterministic. A single-actor test can show you the agent is reasonable. It cannot show you the agent is safe.
I write up more of these production patterns on my blog, if that is useful: https://polycratia.com/c/a82d75f5
Worth knowing where your agent's plan window actually closes: at read, at approval, or at write...
- $ · permalink
Your agent did not get dumber in production. Your infrastructure stopped being synchronous.
In a test harness the agent writes and reads the same store in the same instant: one process, one connection, no lag. So every plan it makes rests on an assumption nobody writes down: what I just did is visible to me right now.
Production slides a read replica, a task queue, and a provider callback between those two steps.
The failure looks like this. The agent calls a tool. The write lands. It reads back to confirm, hits a lagging replica or a job nobody has consumed yet, sees nothing, and decides the action failed. So it retries, which is the sensible move. Now you have two of whatever it was.
And the logs are infuriating, because every single step was correct in isolation.
This is an old problem in new clothes. In payments you learn fast that a transfer which hasn't settled is not a transfer that failed. Treating pending as failed is exactly how you send money twice. The discipline there: the write itself returns the authoritative state and an operation id, and the system models three outcomes instead of two: confirmed, failed, and not yet known.
Agents need that same contract. A tool has to be able to answer "accepted, outcome pending, poll this id", and the agent has to be allowed to wait on that instead of inferring from an empty read. Silence is not a result.
Most agent reliability work goes into prompts and model choice. In my experience the bigger wins sit in the tool layer, where these state semantics actually live. I write up more of these production patterns here: https://polycratia.com/c/c3d0dfcb
If you're running agents against real systems, the thing I'd check first is whether your tool layer exposes a pending state or forces a boolean...
- $ · permalink
The test inputs for my catalog pipeline were a dozen records I had typed by hand: clean fields, nothing merged, no stray markup.
Then production fed it a scraper.
The agent step (translate and adapt a product listing for another market) never failed once. It just got a record where the spec block had collapsed into the description, and it wrote fluent, confident copy around the merged text. Dimensions turned into prose. A packaging note turned into a product feature.
Nothing threw. Some of those outputs read better than the correct ones.
The cost: a batch of published listings describing something that didn't exist, caught by a human reading the storefront, not by anything in the pipeline.
What I had missed is that a model does not error on malformed input. Plausibility is its default behaviour, and broken input comes back as plausible output at exactly the same confidence as good input. In testing I was the input source, so that failure mode had no way to show up.
Now the model boundary has a contract. Fields get validated and normalized before the call (expected units, no markup, every attribute in its own slot), and a record that fails the contract is parked for review instead of passed through. The model gets a narrow job on data I can describe.
I wrote more about the parser-to-model boundary here: https://polycratia.com/c/223cf7b3
If you run a model behind a scraper or an OCR step, the thing worth checking is what happens to a record your pipeline can't parse cleanly: it gets parked, or the model gets to guess...
- $ · permalink
In production your agent didn't fail. It succeeded against a smaller world than it thought it had.
What usually opens the gap is authorization scope, not reasoning quality.
In testing, the agent runs under one credential, almost always the broadest one in the system. Every read returns the whole table, and the plan gets built on a complete picture.
In production it runs on behalf of many principals, and each one sees a different slice. A role-scoped query does not error when the caller lacks access to half the rows: it returns the half they can see, with a 200.
That is what breaks agents specifically. A truncated list is a syntactically valid answer. The agent cannot tell "there are three open items" from "you may see three of the eleven open items", so it reasons confidently on a partial world. Then it closes the wrong thing and reports completion, escalating nothing.
Deterministic code has the same blind spot. The difference: a human reads the resulting screen and senses something is missing. An agent reads it and plans.
How I handle it: scope belongs in the tool contract, not in the infrastructure underneath it. Every read the agent makes returns what was filtered and why. Anything with a non-empty filter reason cannot be the basis for a terminal action, so it routes to a human instead. Permission-shaped emptiness and real emptiness get different types.
It is the same discipline I apply to compliance moderation tooling: the reviewer has to know whether a field is clean or merely invisible to them. Agents need that distinction more, not less, because they do not pause to wonder.
So before you blame the model, run your evals under a narrow principal instead of an admin token. The failures probably show up on the first pass.
I write more about this kind of production detail on my blog: https://polycratia.com/c/01c2c505
If you're running agents against multi-tenant data, go look at whether your tool layer tells the agent what it wasn't allowed to see...
- $ · permalink
I had a test set for an LLM catalog step: a small batch of products I had read end to end, with the output I expected sitting next to each one. It passed. Then it went live behind the parsers, and the thing I had actually been testing disappeared.
In testing I knew the right answer every time. In production that answer does not exist.
The adapted listing comes out fluent and well-formed, and confidently wrong about a dimension. No reviewer sits on the row, and there is no expected value to diff against. Nothing throws. The queue drains. The dashboard is green.
And what broke it was the input, not the model. My test items were clean because I had picked them myself. Real catalog rows come off scrapers with truncated titles, HTML fragments, mixed-language source text, units glued to numbers. The agent did not refuse any of it. It smoothed it over, which I think is the worst behaviour available here.
So I stopped testing outputs and started testing invariants that have to survive the step. Model numbers, size tokens, units, the attribute set: those go in and must come out unchanged, character for character. If a dimension vanishes or a unit shifts, the item goes to quarantine instead of the catalog. The model gets to be creative with prose and with nothing else.
That changed what passing means for me too. The gate before release is now a sample of real production rows, including the ugly ones, and not a curated set I assembled myself.
I wrote up more on how I structure these pipelines on the blog: https://polycratia.com/c/f351a84f
If you run agents in production, I would like to know your detection signal for output that is wrong but well-formed.
- $ · permalink
A locale shipped green: no failed jobs, no errors. Validation passed on every record.
Weeks later someone reading the storefront asked why part of the product copy felt thinner than the rest.
The step was an LLM pass adapting catalog content for an international audience. In testing it had one path: full context in, adapted text out. In production it had two. The provider rate-limited a large batch, my retry wrapper did its job, and after a few attempts the step fell back to a reduced prompt (less context, shorter input) because a degraded result looked better to me than a dead job.
That fallback produced text that was valid. Right language, right fields, length inside the band. It passed every check I had, because every check asked "is this output well-formed" and none of them asked "which path produced it".
So degraded records merged into the catalog, indistinguishable from the good ones. The cost was not an outage: it was not knowing which rows to redo, so I had to re-run a whole slice to be sure.
Now every record carries its own provenance: primary path or fallback, attempt count, model and prompt version. A fallback gets written down as a partial failure that still owes a re-run, not as a success with a quiet asterisk.
Tests exercise the path you designed. Production exercises the paths you added to survive it...
I write more of these production notes on my blog: https://polycratia.com/c/32810fa7
If your agent degrades under load, you probably want to be able to tell, per record, which ones came out of the degraded path.
- $ · 10 min readA date is not a version: strict parsing under OSV ranges
A vulnerability range is a claim about ordering, and it is worth no more than the version parser sitting underneath it. Walk an OSV range's events in whatever order they happen to sit in the file,…
- $ · permalink
Three seconds on an endpoint, and every query inside it runs in milliseconds: the time is not in your code. It's in the wait to get a connection.
Connection pools are the part of a backend almost nobody instruments. A handler reports how long a statement took, not how long it sat queued waiting for a connection to run that statement on. So the slow part is invisible by construction: the query log is clean, the handler looks idle, and the total still lands at three seconds.
In a marketplace backend it looks like this. One endpoint fans out across several models. An export or a reporting query holds a connection far longer than a web request ever should, and the pool is sized to the worker count. Everything else is now standing in line. Wrap an external HTTP call inside a transaction and it gets worse, because the connection stays checked out for the duration of someone else's network.
The fix is not a bigger pool. A bigger pool just moves the queue into the database.
What actually works: measure checkout wait as a first-class metric next to query time. Keep external calls outside transactions. Give long-running analytical work its own pool so it can't starve the request path.
Latency you can't attribute is latency you can't fix. Before you optimize the query, check whether the request was waiting to start it.
I unpack more of these production failure modes on my blog, if that's your kind of reading: https://polycratia.com/c/f7ab9335
Your pool probably gets held hostage somewhere first: exports, transactions wrapping network calls, or something stranger...
- $ · permalink
The 3 seconds weren't the work. They were a poll interval.
The giveaway was the shape of the latency chart, not its height. Response times didn't spread out the way real work does: they clustered on round numbers. One second. Two. Three. Compute doesn't quantize. Waiting does.
The endpoint handed a job to a worker, then blocked on the result backend, asking "done yet?" on a fixed tick. The job itself finished in a fraction of the time the caller spent waiting. The response just sat there until the next tick noticed it had finished. And if the first poll fires before the task even gets picked up off the queue, you've bought a full extra interval for nothing.
That is why standard instrumentation finds nothing. The handler is fast. The worker is fast. The queue sits near empty. Every component reports healthy, because nobody measures the gap between "result written" and "result observed".
Two rules I apply now.
Anything clustering on a round number in a latency distribution I treat as a wait until proven otherwise. Round numbers come from configuration, not from physics.
And synchronous request/response over a task queue is a design smell. Either do the work inline in the handler and own the latency honestly, or return a job id and let the client subscribe to completion. Blocking an HTTP worker on a poll loop pays for concurrency twice and hands you latency you did not choose.
I unpack more of these failure modes in longer form on the blog: https://polycratia.com/c/b668cb9e
I'd like to hear about the most expensive wait you've found that no dashboard was measuring...
- $ · permalink
A moderator hit approve, and the queue took about three seconds to come back. Every time, and only on that one screen.
The search query was fast. The database write was fast. The three seconds sat in between.
The write path pushed the record into Elasticsearch and then waited for the document to become visible before returning, so the reloaded queue would show the item as already handled. That wait has nothing to do with your document or your query: it depends on how busy the index is.
Catalog parsers were bulk-indexing against the same cluster. The one-document moderation write queued up behind their refresh work. Latency on a compliance screen had turned into a side effect of a scraper schedule living in an entirely different service.
What it cost: moderators paced themselves to the UI instead of to the work, and every query-time dashboard looked healthy, because the query was never the slow part. Nobody measured the wait for visibility, only the calls around it.
What I do differently now: read-your-writes does not go through the search index. The endpoint returns the authoritative row from Postgres, indexing happens asynchronously, and the screen reads freshness from the store that owns the record. Search stays with what it is good at, lists and filters, and not with proving that the action you just took landed.
The broader habit out of this one: instrument waits, not just calls. A span around "index the document" would have shown me this in an afternoon. A span around "search" never will.
I write up more of these production post-mortems here: https://polycratia.com/c/0ae66f78
If you run search alongside a write path, I'd look at where your UI gets its freshness guarantee from: the index, or the database.
- $ · permalink
A log line turned out to be the slowest thing in a three-second request.
This was a catalog read path on a cross-border commerce platform. Every number I had looked healthy: Postgres queries in single-digit milliseconds, no queue in the path, cheap serialization, no external API call. Response time still climbed toward three seconds whenever traffic picked up, and it settled back on its own once traffic eased. That is the classic shape of a contention problem, so I spent days on database theories.
The actual cause: structured logs were emitted from a synchronous handler that wrote to a log pipeline over TCP, inside the request. When the pipeline fell behind, the socket write blocked, and the request thread blocked with it. Every log call was quietly a network call to a dependency I had never treated as one.
It stayed invisible that long because I had instrumented the things I expected to be slow. I had timers on query time and cache time, and on outbound HTTP. Log emit was assumed to be free, so nothing timed it, and the gap just showed up as unexplained wall-clock time in the handler.
What I do differently now: logging and metrics go through a bounded in-process buffer with a non-blocking handler, and the buffer is allowed to drop under pressure. Losing log lines during an incident is cheap, and I would rather pay that than serialize user requests behind the log shipper. I time the emit path itself too, because anything that can block in the request path deserves a number next to it.
The broader rule I took from it: observability is a dependency. If it can backpressure, it can take your API down, and it will do that from the one place you are not looking.
I wrote up more of these production debugging patterns on my blog, if that's useful: https://polycratia.com/c/5cbdaa86
One thing I'm curious about, from others who run their own services: do you let your logging path drop data under load, or block to keep it complete?
- $ · permalink
A compliance moderator told me every action in her review queue took about three seconds. The rest of the team said the app felt fine.
The handler timings agreed with them. The view ran its query, serialized the result and returned it, fast and consistent. So I stopped trusting the view and measured where the request entered the process instead.
The three seconds were gone before the handler ran.
Auth middleware introspected the token against the OAuth service on every request, and that service expanded roles into an effective permission set by walking the role hierarchy on each call. Latency scaled with how many roles the user carried, not with the data requested. Regular users had two roles and did not notice. Moderators carried everything, and paid for it on every click.
That is why it stayed invisible: the slow tail belonged to a handful of internal users, the aggregate buried them, and a handler-level timer does not cover code that runs before the handler.
Two changes. The expanded permission set is cached per token now, with a short TTL and explicit invalidation on role change, because permission checks are read-heavy and change rarely. And request timing moved to the edge, so middleware time lands in the same number you already watch.
What I do differently now: when a latency complaint comes from one user and not the aggregate, I look at what is different about that user's identity before I look at the query plan. Permissions are a hot path, and few people treat them like one.
I write these up in longer form on my blog, if that is useful: https://polycratia.com/c/91b83cc2
So it is worth knowing where your request timer actually starts: at the edge, or at the handler.
- $ · permalink
A balance endpoint that stayed fast all day began taking three seconds, but only in the window when the settlement file landed.
No code had changed. CPU was flat. The queue was empty.
The reconciliation importer ran the whole file inside one transaction: parse it, attribute each incoming payment to a user's virtual account, update the ledger, commit at the end. Atomic, easy to reason about. It also held row locks on every account it touched for the full length of the import.
The balance read path took a write lock too: it locked the user's rows to compute a consistent number before returning it. So any user whose account showed up in that file waited behind the importer. Three seconds was just how far the import had gotten.
None of the usual dashboards show this. Handler time, query time and pool stats all looked healthy, because the query was not slow. It was waiting to start.
Three things I now do by default:
The importer commits in small chunks and touches accounts in a deterministic order, so lock windows stay short and deadlocks stop being random.
Read paths don't take write locks. If a read needs consistency, it comes from the ledger's own append-only history, not from locking rows.
Every request path gets a lock timeout. I would rather have a request fail fast with a clear reason than hang and pass itself off as a performance problem.
Missing this costs trust more than latency: a user staring at a stale balance retries the payment, and now reconciliation has a duplicate to explain.
I write these up in longer form on my blog: https://polycratia.com/c/ab86ef71
If you run batch jobs against the same tables your API reads, lock wait is probably worth measuring somewhere, and not only query time...
- $ · 8 min readStuck withdrawals: wait, bump or cancel as a pure function
A withdrawal that has sat unconfirmed for an hour is not in one state. It is in one of three: too early to touch, worth replacing at a higher fee, or past the point where the only useful move left…
- $ · permalink
A webhook handler that returns 200 before the state is committed: one line of code, blast radius measured in days.
Nothing throws. The provider sees success, drops the event from its retry queue, and your system never learns the money moved.
Webhooks are the one place where someone else's retry policy is your durability guarantee. The moment you acknowledge, you own that event. If the acknowledgement sits above the commit boundary (before the transaction, before the queue write), you have quietly turned an at-least-once channel into at-most-once.
From both ends it looks fine. Provider logs say delivered. Your logs show a request served in a few milliseconds. The only trace is a balance that stops matching what actually settled, and that shows up during reconciliation, not during the incident.
The fix is ordering: persist the raw payload with the provider's event id, commit, then acknowledge, then process asynchronously. Deduplicate on that id, because the opposite mistake (doing all the work before replying) buys you gateway timeouts and redelivery instead.
So I treat the acknowledgement line as a boundary worth reviewing explicitly, the same way I review a money-moving transaction. Most small bugs that hurt in production are correct statements sitting in the wrong position relative to a commit.
I write more about attribution, ledgering and reconciliation seams in payment systems here: https://polycratia.com/c/2f4e9d40
Somewhere your system probably assumes a retry it never actually gets...
- $ · 10 min readA 402 is a bill: what an x402 gate owes a client when things fail
An HTTP 402 gate sits between a client's money and a service that can fail on either side of it. The handler can fail. The payment network can fail. Those two failures need opposite answers: a…
- $ · permalink
Most 3-second endpoints are not slow because of the code inside them. They are slow because of everything the response path quietly got made responsible for.
When I open a slow endpoint, the handler itself is rarely the cost. The cost sits in three places.
First, query count. One list endpoint, a nested serializer, and suddenly every row triggers its own lookup. The handler profile looks fine because no single query is slow: there are just hundreds of them. Fixing that is not optimization, it is making data access explicit. Select the relations you serialize, and don't let a serializer decide what to fetch.
Second, synchronous side work. Notifications, audit writes, a provider callback, a document render. Each one is defensible on its own, and none of them belong to the request. In the payment and compliance systems I have shipped, the audit write is the one people are most afraid to move: it feels like it has to happen before you answer. It doesn't. It has to happen durably. Those are different requirements, and conflating them is what puts an external dependency inside your p95.
Third, the response shape. An endpoint that returns everything a client might need pays for everything on every call. Shrinking the payload is usually a bigger win than caching it.
What changes when latency drops that far is the boundary of the request, not the speed of the work. Same work, different place. The queue absorbs what the user never needed to wait for.
And the honest part: the 3 seconds was often never measured as a wait at all. It was measured as handler time, which hides queue wait, connection pool contention, provider timeouts. You cannot fix what your instrumentation attributes to the wrong layer. Time the full path from the client's side first, then decide what to move.
The trade-off is real. Moving work out of the response path means the client gets an answer before the system is fully consistent, and you now owe the user some way to see that. Status fields, idempotent retries, something to poll. That is the actual engineering cost of 200ms, and I think it is worth saying up front rather than discovering it in support tickets.
I write up more of these production trade-offs on my blog: https://polycratia.com/c/b2b0cdbf
For those of you who have done this: what did your instrumentation misattribute the longest, queue wait, pool contention, or an external call hiding inside the handler?
- $ · 8 min readLedger errors should name the invariant they protect
A payout worker catches an exception from the ledger and has to decide, with no human in the loop, whether to retry the request, park it, or escalate. If the refusal says "invalid entry", it cannot…
- $ · permalink
The delivery integration passed every test I had. Then real shipments started coming back with rates the carrier wouldn't honor.
My machine was fine. The problem was that my machine had only ever talked to the carrier's sandbox.
Sandbox responses are curated, with every field present and every status clean, and the calls come back fast. I built my fixtures from those responses, wrote tests against the fixtures, and shipped something that was green end to end against a fiction.
Production sent back a different shape of the same object. Fields the docs called optional were just absent. A rate quote would succeed, the label call behind it would fail, and that left a shipment half-created. Calls timed out and then quietly completed on the carrier's side.
It never looked like an outage. It looked like warehouse staff reconciling shipments by hand, and support answering tracking questions the system couldn't answer. That is what environment drift actually costs you: humans permanently absorbing the gap, with nothing that ever registers as downtime.
So here is what I do now, on every multi-carrier and verification provider integration. I capture and store real production responses as the fixture set. The vendor sandbox I treat as documentation rather than an environment. Every new field shape production reveals becomes a permanent test case, and anything unmatched gets flagged instead of guessed.
I wrote more about how I structure that capture-and-replay layer on my blog: https://polycratia.com/c/28e76d00
If you run provider integrations, I'd be curious what exposed the gap for you first: missing fields, partial failures, or timeouts that later succeeded...
- $ · 10 min readAn EIP-1559 fee ceiling is a hard bound, not advice
An estimator that reads the caller's maximum fee as a suggestion breaks in two quiet ways. It can clamp maxFeePerGas to a value under the current base fee and hand you back a transaction the chain…
- $ · permalink
A fast database rules out one suspect. It tells you nothing about the request.
Query time gets measured inside the engine. Request time is everything around it: waiting for a connection, the number of round trips, serialization, and the part that usually dominates, calls to systems you do not own.
Three mechanics, in the order I usually find them.
Round trips, not query cost. Forty queries at two milliseconds each is a fast database and a slow endpoint. Concurrency makes it worse: each of those forty trips holds a pooled connection, so the pool starts rejecting work long before the engine breaks a sweat.
Inherited tail latency. Shipping rates from several carriers, a verification provider's decision, a quote from an exchange: once any of that sits on the request path, your p99 is the worst p99 in the chain. It is not your code, and no amount of indexing touches it.
Missing timeouts. A lot of HTTP clients default to no timeout at all. Your averages stay clean because the slow path is rare, right up until a provider degrades and requests pile up behind calls that are never coming back.
How I handle it: every endpoint gets an explicit latency budget, and every outbound call declares its share of that budget with a timeout and a defined fallback. Anything not required to produce the response goes to a queue: notifications, label generation, indexing, analytics. And I read spans instead of averages, because averages hide exactly the requests people complain about.
The database is rarely the story. Most of the time I find it sitting on the request path.
I write up more of these failure modes on my blog, if that is useful: https://polycratia.com/c/6d6a50c1
Next time you trace a slow endpoint end to end, I think the answer will surprise you: round trips, a third party, or just waiting on a connection...
- $ · 9 min readReversals are a poll result, not an incident
A deposit that already cleared your confirmation threshold can stop existing. The chain reorganises, the block that carried the transfer gets replaced, and a balance you credited minutes ago is…
- $ · permalink
Rewrite decisions get made about code, and almost every rewrite fails on state.
The five questions worth answering before you touch anything have nothing to do with frameworks.
One: can you run the old and new system side by side on the same data? If the answer is no, you are not rewriting. You are cutting over blind on a date someone picked in a planning meeting.
Two: who owns writes during the transition? Dual-write sounds like a detail until two services both believe they own a balance or an order state. That is where silent divergence starts, and your test suite will not see it.
Three: what is the comparison oracle? On a Python 2 to 3 migration the interesting breakage is not import errors (those surface immediately). It is encoding and numeric behaviour quietly producing different values on the same input. You need a way to replay real traffic through both paths and diff the outputs, otherwise you ship the difference to users and hear about it from them.
Four: which parts are load-bearing for money or compliance? In payments and KYC work, the pieces holding attribution, ledger entries, and verification records are the ones you migrate last, with the most evidence, and not in the same change as a refactor. Everything else can move cheaply.
Five: what do you stop doing while the rewrite is live? A rewrite is two systems in production. Feature work does not pause on its own, and if nobody decides, the team pays the cost twice: once in the old code, once in the new.
My take: the architecture choice is usually the least risky part. The risk sits in the migration path and in write ownership, plus whether you can prove the old and new systems agree on real data. Monolith to services is mostly an exercise in carving ownership of state without losing a transaction.
If the answers are missing, the honest move is strangling the system incrementally (route a slice, verify it, keep the old path warm) rather than a parallel build with a flag day at the end.
I wrote more about how I approach inherited systems here: https://polycratia.com/c/50d88765
For anyone who has done this, you already know which of the five bit you hardest, whether that was the migration, the write ownership, or the feature work that never actually paused.
- $ · 9 min readA ticker is not an asset: USDT at six decimals and USDT at eighteen
Precision is not a property of a ticker. The same stablecoin symbol gets issued by more than one token contract, and those contracts do not agree on how many decimal places the token has: six in…
- $ · permalink
An agent that passes its tests has proven one thing: it can act on a world that stopped moving.
That is the entire gap. A fixture is frozen, so the listing is still available at step 7 because nothing could take it between step 2 and step 7. Production does not hold still. Between the read that informed the plan and the write that carries it out, the balance got spent, the price changed, the order was filled, the document was countersigned by the other party.
The agent does not crash on this. It does something worse: it executes a plan that was correct a few seconds ago, confidently, with a clean log.
Two mechanics make it expensive.
Preconditions get checked once, at planning time, where they are cheap to evaluate and meaningless to guarantee.
And side effects get scattered across the steps, so there is no single place where the world is touched, and so no single place to re-check it before touching.
How I build around it: the planning stage reads and decides, and does nothing else. Every write lands in one executing step that re-validates the preconditions it actually depends on at the moment it acts, against the state version the plan was built on, and it refuses instead of improvising when that version has moved. A refusal you can see beats an adaptation you cannot.
This is not an LLM problem. It is the same discipline payments and marketplace backends have needed for years: conditional writes, explicit state versions, and one owner of side effects. Agents just made it easy to skip, probably because the reasoning step sounds authoritative enough that nobody re-asks the database.
Staging proves correctness under a frozen world. Production asks for correctness under a moving one. Those are not the same test.
I wrote up the mechanics in more depth here: https://polycratia.com/c/1ebc5697
For those of you running agents with real side effects: where did yours first act on state that had already moved, balances, inventory, or document state?
- $ · 10 min readMake a coin selection impossible to construct wrong
The wallet bugs that lose fees do not live in the selection strategy. They live in the seam between that strategy and the transaction builder. You get a selection that overpays by a few thousand…
- $ · permalink
Three seconds to respond, and mostly during the morning peak.
Profiling the handler turned up nothing worth fixing: a handful of queries, all indexed, all fast on their own. So I stopped asking what the request was doing and asked what it was waiting on.
It was waiting on a database connection.
The handler opened a transaction, wrote a verification record, then called the external verification provider over HTTP inside that same transaction. Most of the day the provider answered fast enough that nobody noticed. Under load it answered in seconds, and every one of those requests held a Postgres connection for the whole round trip. Once the pool was saturated, endpoints with nothing to do with verification started timing out too.
The slow handler was the symptom. The shared pool was the blast radius.
What fixed it was ordering, not optimization. Commit the intent first, with an explicit pending state. Make the provider call outside any transaction. Write the result back in a second short transaction, and hand a reconciliation path ownership of whatever never came back.
One rule has been non-negotiable in anything I build since: no third-party network call inside a transaction. Not a slow one, not a fast one. Someone else's p99 becomes your p50 the moment you hold a lock across it.
I write up more production post-mortems like this one on my blog: https://polycratia.com/c/31e1dc5f
Worth checking where your request path still holds a connection open while waiting on someone else's API...
- $ · 10 min readDER SET ordering: enforce it in the walk, not in a check afterwards
A SET in ASN.1 is unordered by definition: the arrangement of its components carries no meaning. DER still fixes exactly one arrangement on the wire, ascending by encoding, compared as octet…
- $ · permalink
In a payment system the most expensive bug is usually a rounding rule nobody wrote down.
Take one repayment split across several investors by share. Every line looks correct, and every line is correct. The sum is not: off by a cent, in a direction that depends on the ratios.
Nothing throws. No test fails either, because the tests assert on a single record, and a single record is fine. The defect exists only in the aggregate, and the aggregate is what the bank settles against.
That asymmetry is what makes it costly. A crash is loud and local. A cent of drift is quiet and compounding: enough distributions later your ledger and the settlement disagree, reconciliation starts flagging rows that are each individually defensible, and you lose the ability to tell drift from something worse. Money systems are append-only, so you don't fix it forward. You correct history, and someone has to decide who absorbed the cent.
So I settle this at design time instead of leaving it to the implementation. Amounts as integer minor units, no floats. The remainder policy gets written out too: largest remainder, assigned to a named party, specified in the spec rather than inherited from whatever rounding mode the language defaults to. Then an invariant at the boundary, where the split has to sum back exactly to the input before anything is persisted.
The general shape is worth keeping. A bug that is wrong only in aggregate is more dangerous than one that crashes, because your test suite and your error tracker are both built to see the second kind.
I write up more of these production trade-offs on my blog, if that's useful: https://polycratia.com/c/ab24a2cb
Worth knowing which part of your system decides the rounding: the schema and the spec, or whichever language touched the number last...
- $ · 10 min readThe mask is a secret too: value barriers in constant-time C++
A secret-dependent choice written as mask arithmetic instead of an if is not yet constant time. If the optimizer can work out what the mask holds at the point you use it, it is allowed to specialise…
- $ · permalink
A three-second endpoint turning into a 200ms endpoint is usually not an optimization story. It's a relocation story.
The code rarely got faster. The work moved out of the request path.
Two places the time actually goes.
Fan-out to systems you don't control. A checkout that asks several carriers for live delivery rates inherits the slowest one, plus every retry on top. Profiling your own handler won't show you any of that. So stop asking at request time: serve from recently refreshed rates, refresh out of band, and keep a narrow synchronous path only for the cases where a stale number is unacceptable.
Computation that belongs to write time. Balances and totals: aggregate state recomputed on every read. Move it to the write, store the result, and the read turns into a lookup.
What makes either of these real engineering instead of a trick is the part the headline leaves out: you now own staleness. Every relocated computation needs an explicit answer for how wrong it is allowed to be and what invalidates it. Also for what happens when the refresh job dies quietly. A cached rate that outlives a carrier price change is a fast response with a wrong price, and wrong costs more than slow.
The other omission: latency is a distribution, not a number. p50 at 200ms while p99 still sits near three seconds means you fixed the common case and kept the one people complain about.
I went deeper on getting work off the request path here: https://polycratia.com/c/5c7150b7
If you have shipped one of these, tell me where the win came from: faster code, or deciding that something didn't need to be computed right then...
- $ · 10 min readA classical handshake is only a finding if you recorded what you offered
No Go repository decides whether its production traffic is protected by a post-quantum key exchange. That gets settled at handshake time, per connection, by the TLS stack inside the binary and by…
- $ · permalink
Docker solved environment parity. It did not touch the thing that actually breaks: state parity.
"Works on my machine" rarely means a missing dependency anymore. It means your machine held one user, one clean dataset, and no history.
Production holds the opposite. A ledger with rows written by three generations of the codebase. Webhooks that arrive twice, out of order, or hours late. Catalog pages from a retailer who changed their markup overnight. Rows written before a column existed, sitting there legally NULL in a field the new code assumes is set.
That does not fit in a fixture.
The failures I keep meeting are the ones real state produces: a payment matched to the wrong obligation because two near-identical requisites existed. A job that passes alone and deadlocks the second two workers run it. A parser that handles the happy path for the top ten SKUs and quietly returns an empty list for the long tail.
So I stopped treating local green as evidence. What I want instead: a production-shaped copy to run against, not a seeded one. Test the second delivery of a webhook, not the first. Run the migration twice. Assume every nullable column is null somewhere (in a system older than three years, it is).
The reframing that helped me most: your laptop is not a smaller production. It is a different system that happens to run the same code.
I write up more of these state-level failure modes here: https://polycratia.com/c/435931fd
Your local environment lies to you somewhere: data shape, concurrency, or time. I think you already know which one...
- $ · 9 min readA strict DER reader needs a calendar, not a clock
The validity period is the only part of a certificate written as text. Everything else DER encodes is bytes with structural rules (a length, a two's complement integer, a base-128 OID arc), and…
- $ · permalink
Your database is fast because it is the only dependency you own. Everything else in the request path belongs to someone else.
That is how a 4ms query and a 900ms endpoint end up sitting happily in the same trace. In the systems I build (multi-carrier delivery, card and account-to-account rails, KYC providers) one request fans out to third parties, and your p99 turns into the sum of other people's p99s, not the max. Three sequential upstream calls at 300ms each: slow endpoint, fast database.
The mechanics that actually bite:
Sequential fan-out when the calls do not depend on each other.
A connection pool sized for the database instead of the slowest upstream, so requests that are waiting hold slots they are not using.
Timeouts longer than the caller's patience, so you keep doing work nobody will read.
Retries stacked on a degraded provider, multiplying load at the exact moment it is failing.
How I approach it: measure time per dependency inside the request, not just the total. Parallelize the independent calls. Set the budget per request, not per call. And move anything not needed to produce the response out of the path entirely, since delivery rates can be precomputed and verification can be asynchronous with a state the client polls.
The part worth keeping is that an endpoint is only as fast as the slowest thing you do not control, and your real lever is deciding what stays in the path at all, not optimization.
I write about this kind of trade-off in more depth here: https://polycratia.com/c/67ce4be5
So your request budget goes somewhere: one slow provider, or five fast ones in a row. Probably worth knowing which before the trace tells you...
- $ · 7 min readThe gap limit is a compatibility contract, not a scanner setting
An HD wallet scanner quits twenty unused addresses after the last one that received money. An address handed out past that window still belongs to your key, your own indexer will watch it happily,…
- $ · permalink
Most rewrite checklists ask about the code. The thing that actually decides how it goes is whether you can run both systems against the same real traffic and diff the results.
The real specification of a legacy system is not in the docs or the tests. It is the behavior it produces in production, bugs included, the ones downstream consumers have quietly built on. Rewrite from the documented intent and you ship something correct and incompatible.
So before I get to the architecture questions, I settle five operational ones.
What is the observable contract: the outputs other systems actually read, not the interfaces you meant to expose.
Which odd behaviors are load-bearing: rounding, ordering, null handling, retry semantics. Some of those are contracts now, whether you intended that or not.
What is the smallest unit you can cut over: one endpoint, one job, one table owner. If the answer is "all of it", you probably have a launch date and not a plan.
How do you go back: not rollback in theory, but a path that still works after the new system has written data.
Who reads the dual-run. Someone has to look at the diffs daily, or shadow traffic just becomes another ignored log stream.
Across Python 2 to 3 migrations and monolith-to-services rewrites of inherited systems, the ones that went quietly were the ones where the old path stayed alive long enough to act as the judge. The ones that hurt were the ones where correctness got argued in review instead of measured against the thing already running.
I write more on legacy modernization and migration strategy here: https://polycratia.com/c/8d8d9fc4
If you've run a cutover recently, I'd like to know what finally convinced you the new path was safe...
- $ · 8 min readRelease a withdrawal hold only when the rail says no
A withdrawal reserves funds before they leave, and something has to let that reservation go when the send does not work out. The trouble is that "did not work out" packs two unrelated facts into one…
- $ · permalink
A worker timed out partway through a batch of product descriptions. The task retried. The catalog came out with two versions of the same item.
Nothing threw. Both runs produced good output.
The pipeline adapts a retail catalog for another locale: source fields in, localized fields out, at catalog scale. In testing I checked the step item by item, feeding a source field, reading the result, judging it. Correct every time. What testing never exercised was the second attempt.
In production that step lives inside a queue. Timeouts, redeploys, a broker hiccup: a retry is normal operations there, not an incident. And the model is not a pure function. Re-adapt the same source field and you get wording that is close but not identical.
Downstream I was keying on the adapted text itself, for dedup and cache lookups, and for change detection on top of those. Close but not identical is a different key. So one retried batch looked like new products and changed products at the same time.
The cost was hours of reconciling a live catalog by hand, plus a stretch where I could not trust my own change feed.
What I do differently now: I treat a model call like a payment request. The output gets persisted before the step completes, keyed by source hash plus prompt version plus target locale, and the step reads through that store. A retry replays the stored result instead of generating a new one. Regeneration becomes an explicit decision, a prompt version bump, and not an accident of infrastructure.
Testing proves the step works. Production asks what happens when it runs twice.
I write up more of these testing-versus-production gaps here: https://polycratia.com/c/b3e44660
I'd still like to know where you pin non-determinism in your pipelines: at the model call, or at the write...
- $ · 8 min readWhy a VEX document should be diffed claim by claim
Nobody reads your VEX document twice. A customer reads the first one you publish, and after that they read the difference between the new one and the copy they already hold: which advisories you…
- $ · permalink
The shipping step at checkout took three seconds, and every profile I pointed at it said my code was sitting idle.
I went in looking for a slow query. There wasn't one.
The endpoint returned delivery options for a cross-border cart, so it had to ask each carrier for a live rate. Those calls ran one after another, inside the request. My own work (validation, normalization, serialization) came in at milliseconds. The rest of the time belonged to whichever carrier happened to be having a bad afternoon. The latency of that endpoint was the sum of other people's infrastructure, and the floor got set by the worst one, not the average.
What made it expensive is where it sat: the shipping step is the moment a person decides whether to finish buying. Three seconds there shows up as abandoned carts rather than a performance ticket. And because each carrier stayed inside its own normal range, nothing alerted. No error, no spike, just a step people quietly walked away from.
What I changed: fan the carrier calls out concurrently, put a hard timeout on each one, and treat a missing quote as a degraded answer rather than a failed request. Show the options that came back, fall back to a stored rate for the one that didn't. Latency becomes the slowest timeout I'm willing to tolerate, instead of an open-ended sum.
The habit I kept from it: when you own an endpoint, you own every dependency's worst case. A call without a timeout is a bet that someone else's system will behave today.
I write these up in longer form, with the reasoning I can't fit in a feed post: https://polycratia.com/c/c92d45f6
If you have an endpoint that fans out to third parties, the thing worth knowing is what it does right now when one of them goes quiet: wait, fail, or degrade.
- $ · 9 min readA constant-time test needs a known leak it is required to find
A timing harness that reports no significant difference has told you one of two things and given you no way to separate them: either the routine under test is constant-time, or the measurement never…
- $ · permalink
A rounding direction is the smallest bug you can put in a money system. It is also one of the hardest to unwind.
Take a repayment that has to be split pro-rata across several investors in one deal: one amount in, many shares out. Each share is a percentage of the total, rounded to the minor unit.
Round them independently and the parts stop summing back to the whole. A fraction of a unit sits unassigned, or one unit too many goes out the door. Every row still reads correctly. Every balance looks plausible. The defect lives in the aggregate, which is where nobody is looking.
Then it compounds. The ledger is no longer closed: each distribution leaves a residue nothing owns. Reconciliation against what the bank actually settled starts disagreeing with your internal balances by amounts too small to trigger an alarm and too persistent to dismiss. By the time someone takes it seriously you cannot recompute the truth, because the rounding was applied at write time and the inputs have moved on.
So I make the remainder an explicit decision instead of a byproduct. Shares get computed in integer minor units, and the residual goes to a named party by a stated rule. Then an invariant asserted before commit: the sum of the parts equals the source amount, or the transaction does not happen.
That check costs nothing to write. Reconstructing six months of drift costs weeks.
The dangerous bugs in financial code do not throw. They are correct in every single record and wrong in the set...
I keep a longer set of ledger invariants I apply to this kind of work on my blog, if it is useful: https://polycratia.com/c/2b05f738
For those of you running distribution logic, I'd want to know where you put the residual, and whether that rule is written down anywhere outside the code. Probably not, in most shops.
- $ · 10 min readA cryptography inventory has to name the module, not just the line
Most of the cryptography your program runs was written by somebody else. A scan that stops at first-party code will faithfully report the SHA-256 on line 137 of a file you wrote and miss the MD5…
- $ · permalink
Cutting an API from 3 seconds to 200ms is rarely optimization. It's a decision about what the endpoint is allowed to wait for.
The 3 seconds rarely belong to your code. In a delivery-heavy marketplace endpoint most of it was fan-out to external providers: carrier rate quotes, one call after another, each on someone else's latency and none of them under my control. Profile the handler and you see almost nothing, because the handler is mostly sitting there waiting.
Three things changed it.
First, the calls stopped being sequential. Rate quotes don't depend on each other, so they run concurrently under one budget for the whole request.
Second, the request got a deadline instead of a timeout per call. When the budget runs out the response returns what it has, and the missing carriers get marked unavailable rather than silently dropped.
Third, everything stable moved out of the request path. Anything that doesn't change per user gets computed ahead of time and read, not derived.
So 200ms is not a faster version of 3 seconds. It's a narrower promise: a bounded answer that may be incomplete, instead of a complete answer with no bound. That trade has to be visible in the response contract, or the client will treat a partial result as the whole truth and quietly show a customer the wrong set of options.
I write up more of these trade-offs on my blog: https://polycratia.com/c/04714806
When you cut latency like this you gave something up, probably completeness, freshness or accuracy...
- $ · 8 min readSign the quote behind every 402, or the retry pays a different price
The x402 exchange comes in two halves with an unbounded gap between them. A server answers an unpaid request with 402 and the terms it accepts; some time later the client repeats the request with an…
- $ · permalink
The matching job ran clean on my laptop, every single time. In production it left a pile of unmatched payments every morning, and someone cleared them by hand.
The service reconciled incoming transfers against expected obligations, grouped by settlement day. Locally everything matched. In production a thin slice of transactions landed on the wrong side of the day boundary and fell into the unmatched bucket.
Same code, same database dump. My laptop and the server just disagreed about when a day starts.
That disagreement was nowhere in the repository. Not in the config file either. It was ambient, inherited from the host, read silently at runtime, never written down anywhere a reviewer would look.
The bug ticket was the cheap part. The expensive part was the daily manual review, and what that does to trust: once a person starts hand fixing reconciliation output, they stop believing the matches that were correct too. You lose the automation long before you lose the data.
What I do differently now: I treat the environment as an input to the program, with the same seriousness as a request payload. Timezone, locale, encoding, decimal precision, all declared explicitly in the application, asserted at startup, and the process refuses to boot if the host disagrees. Then I run the test suite under a deliberately hostile environment (wrong timezone, wrong locale, different collation). If the suite still passes, the code is honest about what it depends on.
"Works on my machine" is rarely about the code. It is about the inputs your code reads without asking: clock, locale, encoding, filesystem case sensitivity, ordering guarantees. None of them show up in the diff.
I write up more of these production post-mortems on my blog, if that is your kind of reading: https://polycratia.com/c/62cf7fab
I'd guess you have one of these too, and it cost more than the ticket said: timezone, encoding, or something ordering-related that only surfaced under load...
- $ · permalink
A fast database doesn't give you a fast API. In most slow endpoints, query time is the smallest number in the trace.
The time goes to everything the handler waits on that isn't your database: a carrier rate lookup, a verification provider check, a payment gateway status call, a file landing in object storage. Each of those is a network hop with a tail latency you don't control and can't tune.
Then fan-out multiplies it. Quote delivery across several carriers one after another and your endpoint's floor becomes the sum of their slow cases, not the average. The database answers in milliseconds: the request spends the rest of its life on someone else's retry.
The other half is that nothing in the request path has a deadline. A client library ships with no timeout, one call hangs, a worker stays occupied, the pool drains, and now requests that touch nothing external are queueing behind requests that do. Latency gets contagious across endpoints that share workers.
How I deal with it: every outbound call gets an explicit timeout smaller than the endpoint's budget, the whole request gets one shared deadline, fan-out runs in parallel instead of sequentially, and anything that doesn't have to be synchronous moves behind a queue with a status the client can poll. Cache what is cacheable. Return partial results instead of blocking on the slowest provider.
The part I'd keep: an endpoint inherits the latency of the slowest thing it waits on synchronously. So profile the waiting, not the querying. A flame graph of your own code won't show you a provider's handshake.
I write up more production trade-offs like this one at https://polycratia.com/c/dbe78c24
Your worst latency probably lives in one of two places: an external dependency, or contention on your own workers...
- $ · permalink
Three weeks into rewriting an inherited service, I found a scheduled job nobody had mentioned, writing straight into the same tables.
Not through the API. Not through any code path I was replacing. A cron entry on a box, doing updates directly.
I had read the whole codebase: endpoints, models, queue consumers, all mapped. None of that told me a second writer existed outside the application I was replacing. My new service was correct in isolation and wrong in production, because it assumed it owned its own data.
That cost me a cutover. Not a catastrophic one, but the kind where records drift into a state neither version expects, and you spend days reconstructing what happened from logs instead of shipping.
So now, before I agree to a rewrite, I ask five things before I open the repo:
Who else writes to this data? Not calls the API. Writes.
What is the oldest record that still has to behave correctly? Legacy systems carry rows written under rules that no longer exist in the code.
How will I prove the new path agrees with the old one, when there is no spec? Usually the answer is running both and comparing outputs on real traffic, not tests written from my own reading of the code.
What has to keep running during the cutover, and can the two versions coexist for a while? If the answer is no, it is not a rewrite, it is a launch with no rollback.
And what happens in the first hour after real traffic hits it?
The pattern I keep seeing in inherited systems: the code is the part you can read, so it feels like the risk. The risk usually sits in the couplings nobody wrote down, a cron job, a report pulling directly from a replica, an integration keyed to a field you were about to rename.
I write up more of these migration and payments lessons here: https://polycratia.com/c/ee005aad
For those of you who have done a rewrite of someone else's system: what hidden dependency did you find too late?
- $ · permalink
Your agent doesn't fail in production because the model got worse. It fails because production won't hold still while it thinks.
A test run is a frozen world: fixtures load, nothing else writes to them, and the agent is the only actor. It reads state, plans five steps, then executes all five against exactly the state it read.
Production works the other way. Between the agent's read and its third action a user cancelled the order, a webhook flipped the payment status, a job closed the accounting period, and another instance of the same agent picked up the same task. The plan was computed against a snapshot that no longer exists. Everything after that is internally consistent and factually wrong.
This isn't an LLM problem. Order books and ledgers have had the same concurrency problem for decades. Agents wrap it in natural language, so it doesn't look like a race condition until money moves or a document gets signed against stale terms.
How I handle it: I treat the agent's plan as a proposal, not a transaction. Each action carries the version of the state it assumed. At commit time the system checks that assumption and rejects the step if the world drifted, and the agent re-plans instead of pushing through. Steps stay small enough that each one commits or fails on its own, so a rejection costs one step and not the whole run.
That turns an unpredictable failure into a boring, observable one: a stale-plan rejection with a version mismatch in the log. Boring failures are the ones you can actually operate.
Most agent evaluation still measures output quality on a static input. Almost none of it measures what happens when the state moves under the agent mid-run, which is the default condition in any system with real users.
I wrote more about how this plays out in payment and marketplace backends here: https://polycratia.com/c/f9162708
If you're running agents against live data, I'd want to know where you put the freshness check: at plan time, before each step, or only at the final write.
- $ · permalink
An endpoint that answers in almost exactly three seconds is rarely slow code. That is a retry budget being spent in full.
Slow code gives you a messy distribution. Work scales with input: more rows, more time. The histogram smears.
Retries do the opposite. Latency goes quantized. You get a cluster near zero, another near one second, another near two, another near three, and almost nothing in between. That shape is not computation. That is a timeout firing, a backoff sleeping, and an attempt starting again, stacked inside a single request the caller believes is one call.
In payment and carrier-rate integrations I have watched this hide for a long time, because each layer on its own looks reasonable. The HTTP client ships with retries enabled by default. The provider SDK adds its own. The service wrapping both has a policy too. Nobody configured a three-second endpoint: three independent one-second decisions composed into one.
The tell is in the shape before it is in the profiler. Stop reading the average, plot the raw distribution, and ask what round number the mass is sitting on. Round numbers come from configuration, not from work.
Two things I now treat as non-negotiable. Retries belong in exactly one layer, and every other layer gets them switched off explicitly. And the total time budget for a request is set at the edge, then passed down as a deadline, so an inner call cannot spend more than the caller is still willing to wait for.
The second one matters more than it sounds. Without a propagated deadline, a retry is just a request that already lost its reader and is still burning a connection from the pool. That is how one slow upstream turns into a saturated service.
I wrote up more of how I approach production systems like this on my blog: https://polycratia.com/c/a6c5253f
The last time you chased a latency number that looked fixed, something was spending it: a retry, a lock, a queue...
- $ · permalink
A repayment landed, got split across the investors who funded that loan, and the sum of the parts did not equal the whole.
Not by much. By the smallest unit the currency has.
P2P lending: one loan is funded by many investors, each holding a pro-rata share, and every repayment has to be divided by those shares and credited to individual virtual accounts. The math is trivial until you round. Each share got rounded to the minor unit on its own, and whatever was left over after the division had nowhere to go.
Nothing failed. No exception, no alert. No failed transaction. Every single distribution was internally consistent and looked correct in isolation. The bank side reconciled fine, because the money genuinely arrived.
The remainder just stayed behind, on the platform's own account, one fraction at a time, across every repayment on every active loan.
That is the part worth remembering: a rounding bug in a distribution doesn't surface as an error. It surfaces as a platform that looks very slightly more profitable than it is, holding money that belongs to investors. It reads like a rounding artifact in a report until someone asks whose money it is, and then it's a compliance conversation.
What I do differently now: distribution is a closing operation, not a loop. The remainder is allocated deterministically to a named recipient by an explicit rule, and the invariant (sum of parts equals the original amount, to the minor unit) is asserted in the same transaction that writes the entries. If it does not hold, nothing commits. The check lives in the write path, not only in a test.
I write more about this kind of money-handling detail on my blog: https://polycratia.com/c/317afccf
If you split money between parties in production, your remainder is going somewhere right now, and you probably want to be able to name the account it sits in...
- $ · permalink
Going from 3 seconds to 200ms is usually relocation, not optimization.
Nothing in the hot path got faster. The work moved out of it.
The endpoints I've seen sitting at multi-second latency rarely have one slow line. They have a response assembled at read time out of things that live somewhere else: a catalog record in one place, availability in another, delivery rates fetched live from several carriers, each call a network round trip with its own timeout budget. Add them up and 3 seconds stops looking like a bug: it's the honest cost of what the handler gets asked to do on every request.
Caching the whole response doesn't save you, because the cache key is usually per user or per destination, and the hit rate collapses exactly when traffic gets real.
What works is splitting the payload by how often it changes. The stable part (normalized catalog data and rate tables, plus anything derived from an ingest you already control) gets computed when the data lands, stored in the shape the endpoint returns. The volatile part shrinks to one indexed read. The handler stops computing and starts looking up.
The trade-off is real: the moment you precompute, you own invalidation. You need a deterministic recompute path, and you need a way to detect drift between the materialized view and the source of truth. Without that you've traded slow-and-correct for fast-and-quietly-wrong, which is a much worse failure mode in anything touching money or delivery promises.
I've written more on where these read paths break in production over on my blog: https://polycratia.com/c/a6fc8858
Last time you cut an endpoint's latency by an order of magnitude, you probably moved the work rather than made it faster, and gave up something to move it...
- $ · 8 min readA cryptography inventory is a reconciliation problem
Most post-quantum migration plans stall on the inventory, before the migration starts. Your source code says one thing about the cryptography you run, the TLS handshake says another, and until both…
- $ · permalink
Docker solved the cheap half of "it works on my machine". The expensive half is still wide open.
The runtime is reproducible now: same image, same interpreter, same pinned dependencies, same OS libraries. That class of bug largely went away, and it deserved to.
What nobody containerized is state.
My local database holds rows I created this morning, in the shape the current code expects. Production holds years of accumulated shapes: payment references truncated by an upstream system, partial amounts against an obligation that was supposed to arrive in one transfer, duplicates from a client retry, rows written under a schema two migrations ago and never backfilled.
The code path runs identically in both places. The outcome differs because the input differs.
Same pattern outside payments. A catalog parser works perfectly against the page I opened while writing it, then it meets a regional markup variant, a product rendered in a different template, a listing that is technically valid and structurally nothing like my fixture.
So the honest version of the phrase is not "my machine is different". It is "my data is younger".
What I do about it: I treat production data shapes as first-class test fixtures. The ugly cases (unmatched payments, malformed requisites, odd catalog pages) go into the suite as permanent fixtures, anonymized. I make the code state its assumptions explicitly instead of holding them silently. And when something will not reproduce locally, I stop hunting for a version difference and start asking which row exists there that does not exist here.
Environment parity is a solved problem. State parity is a design decision, and I think most teams never make it.
I go deeper into how I structure this kind of work on my blog: https://polycratia.com/c/753bd780
When a bug refuses to reproduce locally, what do you check first: versions, data shape, or event ordering?
- $ · 9 min readWhy leniency in a DER parser becomes a signature bypass
Two implementations can read the same X.509 certificate and disagree about what it says. The cause is almost never the cryptography. BER permits several encodings of one value, DER permits exactly…
- $ · permalink
A checkout page started timing out, and I spent the first hour staring at the database.
Every query in the trace came back fast. Indexes were fine. The connection pool sat nowhere near saturated. The slow part wasn't in Postgres at all: it was the shipping step.
That platform quotes delivery across several carriers, and the handler asked each carrier for rates, one after another, inside the request. Nobody noticed for a long time, because the calls were quick and their sum stayed under the threshold where a page still feels alive. Then one carrier's API started answering slowly. Not failing. Just slowly.
Failures you catch immediately. Slow success looks like health.
What it cost wasn't a broken checkout, it was worse: a checkout that worked, took far too long, and pushed people into refreshing and re-submitting the same order. Duplicate submissions, support traffic, and a long argument about database performance that had nothing to do with the database.
What I changed, and still do:
I measure time spent inside my own process separately from time spent waiting on someone else's, in the same trace, as two different numbers. If those are one number, you will misdiagnose every incident like this one.
Every outbound call in a request path gets its own timeout, set well below what the client will tolerate.
Fan-outs go concurrent, and the response degrades to partial results instead of waiting around for the slowest participant.
Anything not required to answer the request leaves the request path entirely: quotes that can be pre-warmed, label generation, notifications. None of that belongs between the user and their response.
A fast database only tells you your database is fast. The request is as slow as the slowest thing your handler is willing to wait for, and most handlers will wait forever unless you tell them otherwise.
I write up more of these production post-mortems here: https://polycratia.com/c/9910a389
Next time you profile your slowest endpoint, look at how much of that time was your own code actually running. Probably less than you'd guess...
- $ · 8 min readConstant-time comparison and erasure the compiler cannot delete
Two defects outlive most of the code that still contains them. A MAC compared with memcmp returns at the first differing byte, so anyone who can time the call learns how many leading bytes they…
- $ · permalink
The ugliest function in the payment service I inherited had eleven branches and no comments, plus a variable called tmp2.
I wanted to delete it on day one.
The plan looked reasonable: finish the Python 2 to 3 migration, then carve the payment attribution layer out of the monolith and rewrite it properly. Normalized requisites, deterministic matching, clean code. I had built that layer before, from scratch, more than once. I knew what good looked like.
The migration went fine. The rewrite went fine too, right up until real money started moving through it.
Then unmatched payments started piling up in the exception queue. Not many at first. Enough that someone had to work them by hand every morning.
Every one of them traced back to a branch I had removed.
A reference field that arrived truncated from one bank and full from another. A partial payment that had to attach to an existing obligation instead of opening a new one. A duplicate that was a genuine second transfer, not a retry. None of that was in any spec. It was in the code, because someone before me had learned it the expensive way and encoded it where it would actually run.
I put the branches back one at a time, each with a test named after the failure it prevents. The function is still ugly. It is now also documentation.
So when someone asks me what to ask before a rewrite, I have a list of five questions, and one of them outranks the rest: what does this code know that no document does?
The way I answer it now is boring and it works. Before I touch anything, I instrument the old path. Log which branch fires, how often, and on what input. Let it run through a full billing cycle, because the interesting cases are monthly, not daily. Branches that never fire are dead and I delete them with a clear conscience. Branches that fire are requirements, and I owe each one a test before it disappears.
A rewrite is not a chance to be right from first principles. It is a migration of knowledge, most of it undocumented, some of it load-bearing, and all of it yours the moment you merge.
I write more about working inside inherited systems here: https://polycratia.com/c/bc9dbe56
If you have inherited something you badly wanted to rewrite, you probably already know which piece of the mess turned out to be load-bearing.
- $ · 8 min readAn address validator should return a reason, not a boolean
A user pastes an address into a withdrawal form and gets back "invalid address". Sometimes the string was truncated by a copy that clipped at a line break. Sometimes it is a perfectly good address…
- $ · permalink
The agent passed every sample I fed it. Then it went live on the full catalog and started quietly rewriting product units.
I was running an LLM pipeline that adapts retail listings for another market: titles, descriptions, category wording. In testing I pulled a sample, read the outputs myself, and they were genuinely good. Right tone, real nuance, far better than a translation API.
In production the same prompt met listings I had never read. Somewhere in a long description the model decided "pack of 6" belonged in the title and rewrote the unit. It rounded a weight. It reflowed a dimension string into prose, because prose read better.
None of that is a translation error. But weight and dimensions feed carrier rate calculation, and the unit feeds price, so a language task silently became a pricing and shipping task. Re-running the prompt was cheap. The expensive part was re-deriving downstream state that had already been published.
What I do differently now: the model never writes to the system of record. It writes proposals into a staging table, and a deterministic validator decides what gets promoted. Numbers, units, currency, dimensions, SKU have to come out identical to the input. Only free text is allowed to differ. Anything touching a structured field is rejected outright, not sent for review.
The model doesn't have to be right about everything. It has to have a blast radius small enough that being wrong is boring.
Longer write-up here: https://polycratia.com/c/646bb70a
If you run an LLM inside a pipeline, I'd look hard at whatever sits between its output and your system of record.
- $ · 9 min readThe only stateful part of an x402 payment gate is the replay ledger
An x402 paywall is three steps: the server answers an unpaid request with 402 and the terms it accepts, the client repeats the same request carrying an X-PAYMENT header, and the server verifies,…
- $ · permalink
A compliance reviewer told me the account page took about three seconds to open. On my machine it was instant.
The difference was the test data. My fixture account had a handful of transactions. Hers had years of them.
The balance was computed on read: sum every ledger entry for that user, every time the page loaded. It was correct. It was honest too. And it got slower with every payment the account had ever made.
The endpoint was not slow in general. It was slow in proportion to how much the customer had used it. The accounts that mattered most were the worst ones to work with, and those were exactly the accounts reviewers opened all day.
The expensive part: nobody filed it as a bug for months. It didn't read like a defect, it read like a heavy page. People learned to wait, and the moderation queue quietly absorbed the delay.
Now I treat a balance as stored state instead of a query. Entries append and the stored balance moves with them inside the same transaction. A separate job recomputes from history to prove the stored number still matches. Read paths stay off the past.
I also stopped benchmarking against fresh fixtures. If an endpoint touches a user's history, I test it with the oldest, busiest account in the system, the one that will exist in two years rather than the one created five minutes ago.
I write more about ledger and reconciliation trade-offs here: https://polycratia.com/c/719c4fe4
Worth knowing where your data grows fastest per user, and whether there's still a read path that walks it end to end...
- $ · 8 min readThreshold approvals belong on the request, not on the button
Above a configured amount, a withdrawal should not leave the system because somebody clicked approve. It leaves because enough distinct people have signed that specific request, and because those…
- $ · permalink
The payment had landed. The money was sitting in the account. The user's balance still said zero.
That is how I found out my normalization function was eating leading zeros.
Incoming transfers arrive tagged only by requisites: an amount, a reference string, a timestamp. To attribute them you normalize the reference first, trim whitespace, drop separators, uppercase. Somewhere in that chain I was also casting the reference to an integer before comparing it. Reference 0041 and reference 41 became the same key.
For a long time it did not matter. A collision needs two live obligations whose references differ only by padding. Then one day there were two.
The bad match itself was small. What made it expensive was that nothing failed: no exception, no unmatched-payment alert. The matcher was confident. Aggregate reconciliation against the bank came back clean, because every cent was accounted for, it was just filed under the wrong person. It surfaced only when a user complained about their balance, and by then days of ledger entries had been written on top of the bad attribution.
Two things changed permanently after that.
Normalization became a pure, versioned function with its own test suite, and I now keep the raw requisite stored next to the normalized key forever, so any past match can be re-derived and audited instead of argued about.
And the matcher is no longer allowed to be clever. If a payment matches more than one obligation, it does not score them and pick a winner: it refuses and routes to manual review. Unmatched is a cheap problem, someone looks at a queue. Mis-matched is an expensive one, because it hides inside a total that still balances.
I write up more of these payments and reconciliation lessons on my blog, if that is your kind of reading: https://polycratia.com/c/29fc851e
If you run attribution today, the thing worth checking is what your matcher does when two candidates tie: fail loudly, or quietly choose...
- $ · permalink
Cutting an API from three seconds to 200 milliseconds is rarely a performance win. Usually it is an admission that the endpoint was doing someone else's work synchronously, on your caller's clock.
Here is the shape it takes in the systems I build. A request arrives, and inside the handler it fans out to third parties: carrier rate quotes, then a verification provider, then a payment status lookup on top of that. Each call is acceptable on its own. Serialized inside one request they stack, and your p95 now belongs to the slowest external party on their worst day.
No amount of query tuning fixes that. Indexes do not make someone else's API faster.
The fix is changing what the endpoint promises. Pull the fan-out out of the request path. Refresh carrier rates on a schedule and serve them from a local table with an explicit freshness stamp. Turn "go ask the provider for status" into "read the status recorded when their webhook arrived". Return what you already own, not what you have to go fetch.
That trade has a price, and it is worth naming. Two things keep it honest.
The response has to carry how old the data is, so callers can decide whether stale is acceptable for their use case. A shipping estimate tolerates minutes. A compliance check usually does not.
The write path has to be idempotent, because you are now recording state from retries and webhooks that will arrive more than once, out of order, and occasionally contradicting each other.
Miss the first and you serve wrong data quickly and quietly. Miss the second and you get duplicate state that surfaces weeks later in reconciliation, which is a much more expensive bug than a slow endpoint.
The latency number is the side effect. The real change is that your endpoint stops failing when a third party has a bad afternoon: you degrade to slightly older data instead of timing out.
I have written up more of these production trade-offs on my blog: https://polycratia.com/c/38d85f48
If you have made this move, the first thing that broke was probably one of two things: readers who needed fresher data than you assumed, or duplicate writes from webhook retries...
- $ · permalink
"It works on my machine" is almost never a packaging problem anymore. Containers closed that gap years ago.
What survives is that your machine is single tenant and synchronous.
Locally one request runs at a time. Nothing else touches the row you just wrote. The callback shows up after the response you were sitting there waiting for. Latency is effectively zero, so ordering looks like a property of your code instead of a race you happened to win.
Production runs the other way. Two workers pull the same job in the same second. A provider callback lands before your own transaction has committed. A retry arrives while the first attempt is still open. None of that is a config difference, and no amount of environment parity reproduces it.
The third party side is worse, because it looks solved. Sandbox endpoints answer instantly and deterministically. Real KYC providers return pending and flip status later, sometimes out of order, sometimes twice. Payment rails settle on their own schedule, and the settlement you reconcile against is not the response you got at the API edge. Your local run just doesn't visit the states that decide whether your state machine is correct.
So the question I ask about a piece of code isn't "does it run". It's this: what happens if this executes twice in the same instant, what happens if the callback arrives before the write, what happens if it never arrives at all. Those answers are cheap at design time and very expensive once money has moved through the path.
Environment parity is still worth having, I think. But understand what it buys you: determinism. And determinism is precisely the property production doesn't have.
I write up more of these local versus production failure modes here: https://polycratia.com/c/3cd1fc50
When local passes, staging passes, and production still breaks, what do you check first: concurrency, ordering, or the third party?
- $ · permalink
A fast database does not give you a fast API. Query time is the slice of the latency budget you actually control, and most of that budget belongs to systems you don't own.
Look at where a request in a payments or marketplace backend spends its time. A checkout call quotes rates from several carriers. A signup call waits on a KYC provider's decision. A payout call waits on a payment provider that is quick on a good day and tells you nothing on a bad one.
Each of those is a network round trip to an endpoint with its own load and its own queueing, plus an incident you will never be told about.
Serialize three of them and your median is the sum. Your tail is whichever dependency is having a bad afternoon.
Then the blocking spreads. A worker waiting on an external response still holds its slot, and often still holds its database connection. One slow provider quietly turns into a globally slow API, while the database dashboard stays green: which is probably why people keep optimizing the one layer that was never the problem.
The fix is a boundary, and no index will get you one. Decide explicitly what the request path is allowed to wait on. Everything else gets a timeout you chose, or a fallback answer, or a pending state that reconciles asynchronously. Cache what is stable: carrier rates change far more slowly than checkouts happen. And measure latency per dependency instead of as one number, or you average the problem away.
I write up more production trade-offs like this one at https://polycratia.com/c/4cbf07a8
Worth knowing where the time actually went, the last time you chased a slow endpoint...
- $ · permalink
A rewrite is rarely a code problem. It is a problem of everything the code does that nobody wrote down.
Five questions decide it before the first commit.
What behaviour is load-bearing but unintentional? Old systems collect quirks, and downstream consumers have already adapted to them. Rounding that falls one way every time. A field that arrives empty instead of null. You can reproduce it or you can break it, but decide that on purpose rather than find out in production.
Where does state live outside the database? Queue messages in flight, cron runs that are half done, retry counters sitting in a worker. A cutover moves rows. What is mid-flight it hardly ever moves.
Who calls this that you do not control? Integrations you cannot version on your own schedule: their change window sets your timeline, not the other way round.
Can both systems run at once and be compared? If you cannot push real traffic through the old path and the new one and diff the outputs, you are not migrating, you are hoping. Reconciliation is the migration.
What is the unit of rollback? One endpoint, one service, one table. If the honest answer is "the whole thing", your plan has no brakes.
In Python 2 to 3 migrations and monolith-to-services splits the failures cluster in the same place: not the new code, but the assumptions the old code was quietly satisfying for years. The new implementation is usually correct. Correct against a specification nobody ever agreed to...
I wrote more on how I approach cutovers like this: https://polycratia.com/c/ef123070
Of those five, one gets skipped most often in the rewrites you have been part of.
- $ · 9 min readParse crypto amounts like hostile input, declare rounding first
Every crypto amount enters a system as a string: a form field, a JSON body from an exchange API, a row in a payout file, a webhook from a provider. Between that string and the ledger two decisions…
- $ · permalink
An AI agent that works in testing and fails in production usually did not get worse at reasoning. It just met a second copy of itself.
In testing there is one agent, running one task, in one clean sequence. In production that same agent runs concurrently across users, retries itself on a timeout, and gets replayed by whatever queue sits in front of it. Now two runs are touching the same external state at the same time, and that state lives outside your database: in a payment provider, a CRM, a mailbox, a document workflow. You cannot wrap it in a transaction and roll it back.
This is not a new class of bug. It is the same failure mode I have been designing around in payments since 2018: an action that is safe once and destructive twice. The difference is that a payment client retries on a fixed rule you wrote. An agent decides to retry on its own, with slightly different wording, and your dedup key never matches.
What I do about it is boring, and it works. Every side effect an agent can trigger gets an operation record before the call, keyed on the business fact (this user, this invoice, this document) and not on the agent's generated text. The tool checks that record first. If the operation already exists, it returns the prior result instead of acting again. The agent is allowed to be non-deterministic. The effect layer is not.
The second thing: the tool layer refuses, loudly, rather than improvising. A tool that returns an empty result on failure teaches the agent that nothing was there, and the agent moves on. Absence and failure have to be different return values, or the model will confidently reason from a lie.
My take: reliability of an agent is not a property of the model. It is a property of the boundary you put between the model and anything irreversible. Test suites almost never exercise that boundary, because the boundary only breaks under concurrency and partial failure, which is exactly what staging does not have.
I have written more on designing those effect boundaries here: https://polycratia.com/c/6c4a27b7
If you run agents against real external systems, something already forced your tool layer to be idempotent: duplicate writes, duplicate charges, or probably a duplicate message that went out to a customer...
- $ · 7 min readIssuing deposit addresses without holding a private key
A custodial service that shows a user a deposit address has to answer a question the user does not ask: which process produced that address, and what else can that process do? If the answer is that…
- $ · permalink
That endpoint was fine for almost everyone. It was slow only for the accounts carrying the most data.
A 3-second response usually gets read as a code problem: profile the handler, add an index, move on.
But the median looked fine. So did the 90th percentile. The slowness sat in a thin tail, and everyone in that tail had one thing in common: unusually large related data. On a lending deal that meant many investors sharing one loan, each with a position to resolve. In a catalog it meant one product with hundreds of variants.
The handler did a fixed amount of work per related row. Cheap at five rows. Ruinous at five hundred. Latency was never a property of the code, it was a function of the data shape the caller happened to have.
Which is why dashboards missed it. They average across users whose data does not look alike, and the shape that breaks you is a minority by definition.
So now I log cardinality next to duration: how many rows, children or positions a response actually assembled. Then I plot latency against size instead of against time. A flat line means fixed cost. A slope means per-row work you will meet again, larger, later.
The slope is the bug. The seconds are only the symptom.
The uncomfortable part is that this failure mode scales with success. Your heaviest, longest-tenured, most valuable accounts hit it first, and they are the ones least likely to file a support ticket about it.
I wrote up more of these slow-response post-mortems, including the ones that hid the longest, here: https://polycratia.com/c/682ba0de
For you it might have been a specific tenant, or a payload size, or a dimension you were not logging at all...
- $ · 7 min readERC-20 amounts are meaningless without the token's decimals
An ERC-20 balance is a uint256 counting the token's smallest unit, and nothing in that integer tells you how many units make one token. You have to ask the contract. Assume eighteen decimals against…
- $ · permalink
Every dependency in the request was healthy. The endpoint was still slow.
Both of those were true at the same time.
This is where a lot of latency work goes wrong. You open the database dashboard: queries are fine. You open each downstream service, and each one reports a reasonable average. Nothing looks broken, and users still wait.
A user request does not experience averages. It experiences the slowest thing on its own critical path.
I hit this over and over on shipping and checkout endpoints in cross-border commerce. One call would read the catalog, then fan out to several carriers for live rates, then touch payment and currency logic before it could answer anything. Each integration was fine on its own. But once a single request depends on six things, the chance that at least one of them is having a slow moment stops being an edge case. The tail latency of that endpoint ends up much closer to the maximum of its calls than to any average you happen to be staring at.
So I stopped measuring components and started measuring the request. One timeline per request, every external call on it, ordered, waiting time visible. That view usually answers the question in minutes, and what it turns up is almost always something nobody suspected: a serial chain of calls that could have run in parallel, or a retry policy quietly tripling one leg, or an enrichment step nobody needs before the response goes out.
The fixes are unglamorous. Give every external call an explicit time budget instead of the default timeout it inherited from some library. Run independent calls concurrently instead of chaining them. And decide up front what the response looks like when a dependency does not answer in time, because a degraded answer (a cached rate, a narrower list) beats holding a request hostage to the slowest participant.
Latency is a property of the composition, not of the parts. That is why a fast database and a slow API are not a contradiction.
I write up more of these production trade-offs here: https://polycratia.com/c/7234d751
The last time you chased a slow endpoint, the time was probably hiding somewhere you weren't looking...
- $ · 7 min readIdempotency belongs in the ledger, not in every caller
A payment callback that arrives twice must not move money twice. The usual answer is a unique constraint on the entry id, which turns the second attempt into an error, and an error is not the same…
- $ · permalink
Most KYC integrations don't fail on the happy path. They fail on the states nobody modelled.
Teams treat verification as a gate: the user submits documents, the provider answers, you set verified = true and move on. That works until production.
Then you meet the middle. A provider returns pending and never calls back. A document expires while the account is live. A manual reviewer wants more information. A user who passed six months ago trips a sanctions re-screen. None of that is approved or rejected, and a boolean has nowhere to put it.
What I do instead: model verification as its own state machine with explicit intermediate states, each one persisted with the reason it was entered and who or what can move it forward. Provider responses are events applied to that state, not the state itself. Re-verification is a normal transition, not an exception.
The practical payoff is the moderation queue. A compliance reviewer opens a case and sees the current state plus the justification for it, instead of reconstructing history from provider logs. And when a user is stuck, you can answer why in one query.
The other benefit: onramps and payouts get to ask one question, is this user permitted to do this action right now, rather than every service inventing its own reading of a flag.
If you run KYC in production, there's probably one intermediate state that cost you real money before you gave it a name...
- $ · 8 min readunder_investigation is a status, not an excuse
A scanner tells you a component in your build carries a known advisory. It cannot tell you whether the vulnerable code is reachable in your product, and that judgement is the entire content of a VEX…
- $ · permalink
not_affected is a claim you make on someone else's behalf. under_investigation is usually the honest answer.
The failure mode is quiet. A finding lands, nobody can prove the vulnerable path is actually reachable, and it gets closed as not_affected because that clears the board. Unknown reachability just became a negative assertion. The backlog didn't shrink — it turned into assurance a downstream consumer will rely on.
I have hit this exact shape in payment attribution. Funds arrive, they don't match cleanly, and the tempting move is to guess the owner so the exception queue stays empty. The discipline that holds is the opposite: never guess, flag the unmatched, and treat "not resolved yet" as a real state with its own review path — not a gap you paper over.
Same rule for VEX. under_investigation is a status, not an excuse. And any status that survives review has to carry its why, because a verdict without a justification can't be re-checked when the code moves underneath it.
Where does your triage quietly convert unknown into no?
https://polycratia.com/c/39307b9c
- $ · 8 min readThe ERC-20 that returns nothing still moved your money
Three behaviours in deployed token contracts break integrations that trust the return value of a transfer: contracts that return no data at all, contracts that refuse to raise an allowance that is…
- $ · permalink
Most money bugs I have chased were not float bugs. They were rounding decisions nobody made.
The usual story: an amount shows up as a string from an API, a webhook, a user form. Somebody parses it into whatever numeric type is nearest to hand. Then it goes through a conversion, a fee, a split across investors, a display layer. Every one of those steps rounds, and none of them was asked to. The final number is whatever the last operation happened to leave behind, and three weeks later during reconciliation it is off by a unit nobody can explain.
So I treat inbound amounts the way I treat any hostile input.
Parse from the string, never from a float. The string is what the counterparty actually sent; a float is already a lossy reading of it.
Validate against the asset's own precision. Eighteen decimals is normal on one chain and nonsense on a card rail. If a value carries more digits than the asset can represent, you do not quietly truncate it, you reject it at the boundary. Truncating means you have silently accepted an obligation you cannot settle exactly.
And make rounding a declared policy, checked before the arithmetic runs, not a side effect you find afterwards. Half-up, floor, banker's: the specific choice matters far less than the fact that it was chosen explicitly, is visible in the code, and is the same on the ledger side as on the display side.
That is the thinking behind cryptomoney, a small open-source library I built for exactly this: precision-aware parsing, explicit rounding policy, and loud failure on anything that does not fit the asset.
The deeper point is architectural. Rounding is a business rule. Leave it implicit and it does not disappear, it gets spread across every function that touches an amount, and each one gets a vote. Ledger drift is rarely one dramatic bug. It is a hundred small unowned decisions agreeing to disagree...
When an amount crosses a boundary (chain, rail, service, currency) precision and rounding are part of its type, not a formatting concern for later.
I wrote up more of the design reasoning on the blog: https://polycratia.com/c/36032cb1
If you run multi-asset ledgers, you enforce precision somewhere: at ingestion, at the type, or at the database column. I have seen all three, and the failure modes are very different.
- $ · permalink
A token transfer that returns nothing can still have moved your money. And a transfer that returns true can still have moved less than you asked for.
ERC-20 is a standard the way a handshake is a standard. Plenty of deployed contracts leave out the boolean return, so a strict integration reverts on a transfer that actually went through. Some refuse an allowance increase unless you set it to zero first, so your approve call fails against a balance that is sitting right there. Some take a fee out of the transfer, so the amount you credited internally is bigger than the amount that arrived.
Each one breaks a different layer. The first breaks your call site. The second breaks your onboarding flow. The third breaks your ledger quietly, and that is the one you find weeks later during reconciliation.
The habit that survives all three: don't trust the return value as evidence of the effect. Read the recipient balance before, read it after, credit the difference. The transfer is a request, the balance delta is the fact.
I wrote it up properly here: https://polycratia.com/c/047ca8ee
The missing return, the approve reset, the fee on transfer: you have probably met at least one of them in production...
- $ · 8 min readA withdrawal is a queue entry before it is a transfer
Most withdrawal code collapses two different events into one function call: the user asking for money to leave, and the money actually leaving. Once those are the same operation, everything you…
- $ · permalink
The service that hands out deposit addresses should be incapable of spending from them.
Not "trusted not to". Incapable.
That is the whole reason to derive deposit addresses from an account level extended public key. You hand the address service an xpub and an index. It derives the next address and gives it to a user. It holds no private key, so there is nothing on that box for an attacker to steal and move funds with. Signing sits somewhere else entirely, on a different trust boundary, usually touched by a different process and different people.
I have built custodial wallets for BTC and ETH, and this is the design decision that ages best. Most other key handling policy I have seen is a promise you keep by discipline. This one is kept by arithmetic: public derivation cannot produce the private half.
What surprises people is where the risk goes once you do this.
It goes into bookkeeping. Two things now decide whether a deposit is ever seen: the derivation path and the gap limit.
The derivation path is your account structure. Get it wrong once, in a migration or a rewrite, and you end up watching a perfectly valid branch of a tree that no user was ever given an address from. Funds land at addresses nobody is looking at. The chain is fine. Your ledger says nothing.
The gap limit is worse, because it fails quietly and only under real usage. You scan a window of unused addresses ahead of the last one you saw money on. Hand out addresses faster than deposits arrive (bots, abandoned checkouts, users who generate and never fund) and the funded address drifts past the end of that window. The deposit is confirmed on chain and invisible to you. Nobody gets an error. Support just gets a message saying "I sent it, where is it".
So I treat the watch set as a first class piece of state, not a derived convenience: every address handed out is recorded at issue time with its full path, and the watcher runs off that record rather than an optimistic scan window. The gap limit becomes a fallback for recovery, not the source of truth.
This came out of chain-addresses, a small open source project of mine for deriving and tracking deposit addresses. The longer write up on derivation paths and gap limit failure modes is on my blog: https://polycratia.com/c/d5daf33e
If you run custodial deposits in production: do you drive your watcher from issued addresses, or from a scan window over the xpub?
- $ · permalink
The fastest way to lose control of a custodial wallet, I think, is to model a withdrawal as a transfer.
A transfer is a single moment. It either happened on-chain or it didn't. If that is your model, then the instant a user hits withdraw you are already committed. There is no safe point to cancel. No window to batch several outputs into one transaction either, and no chance to re-price the fee when the network moves under you.
A withdrawal request is not a moment. It is a record with a lifecycle: requested, held for review, approved, queued, signed, broadcast, confirmed. Every one of those is a place where a human can still step in, or a risk rule, or a fee policy.
I have built custodial BTC and ETH wallets and on-chain payment rails, and the pattern holds every time. Once the request is its own row with an explicit state machine, broadcasting becomes one transition inside the operation rather than the entire operation. Cancellation becomes a state change instead of an apology. Batching becomes a scheduling decision. Compliance holds stop being special-cased hacks bolted onto the send path.
Infer the state from the chain instead, and you get a system that can only tell you what already happened...
Wrote up the full argument here: https://polycratia.com/c/4298b9da
If you run withdrawals in production, I'd like to know what forced you to add the request record: a cancellation you couldn't honour, or probably a fee spike you couldn't re-price.
- $ · 8 min readA ledger should not own a money type
Most ledger libraries ship their own Money class, so any system that already had a money type ends up with two of them and a conversion layer in between. A ledger does not need to own a money type.…
- $ · permalink
The moment a ledger ships its own Money class, your system has two money types.
One from your currency library. One from the ledger. Every posting becomes a conversion at the boundary, and every conversion is a place where scale, rounding, or currency can quietly change.
In the payment systems I have built — per-user account ledgering, attribution by requisites, reconciliation against what the bank actually settled — the bugs that take longest to find are almost never in the arithmetic. They live in the translation layer nobody thinks of as logic.
So I stopped letting the ledger define money. A ledger needs exactly three things from a value: an exact amount, a currency, and arithmetic that obeys the rules of that currency. That is a protocol, not a class. Anything that satisfies it can be posted.
One money type per system, owned by the currency library, consumed by the ledger.
The trade-off is honest: you give up having the ledger validate money for you, and you take on making your own type correct. I would rather get one type right than keep two in sync forever.
Where does your system convert between money representations today — and do you know every place it happens?
https://polycratia.com/c/c9290dd5
- $ · permalink
The chain can tell you a transaction confirmed. What it can't tell you: why someone asked for it, who approved it, or whether it should have gone out at all.
That gap is why I model a withdrawal as its own record before it is a transfer.
Requested, approved, signed, broadcast, confirmed: explicit states, explicit transitions, each one written down when it happens, not reconstructed later by scanning the chain.
The payoff is everything that happens before broadcast. While a withdrawal is still a request, you can cancel it, hold it for review, reprice it, or batch it with others heading to the same place. After broadcast none of that exists. You have one irreversible fact and a support queue.
Inferring state from the chain also erases the transitions that matter for audit. A confirmed transfer looks identical whether a human approved it, it was auto-approved under a limit, or it was retried after a stuck broadcast. Three different events, three different consequences, and only the record keeps them apart.
I've rebuilt this shape enough times across custodial wallets, exchange payouts and stablecoin rails that I turned it into an open-source project. I called it withdrawals, because the state machine barely changes between them.
Longer write-ups on payments and crypto rails live here: https://polycratia.com/c/60317e09
If you run custodial payouts, I think the interesting part is where you draw the auto-approval line: amount, destination history, or something else...
- $ · 9 min readFee estimation belongs inside the coin selection loop
Coin selection on a UTXO chain is usually written as two steps: pick inputs until they cover the amount, then compute the fee and adjust. That order cannot be made to work. The fee is a function of…
- $ · permalink
The fee for a Bitcoin withdrawal depends on inputs you haven't chosen yet.
That sentence sounds like a paradox, and in coin selection it is a genuine loop. The fee is a function of transaction size. Size is a function of how many inputs you spend. And how many inputs you need to spend depends on the fee, because the fee comes out of the same funds.
The usual implementation pretends the loop isn't there. Select inputs to cover the amount, then bolt the fee on afterwards. Now the selected set no longer covers amount plus fee, so you pull in one more input, which makes the transaction larger, which raises the fee again. Or the code quietly resolves it by shrinking the change output until it drops below dust — and that change is not returned to the user, it's handed to miners.
When I built custodial BTC wallets, the fix was structural, not arithmetic: every candidate input set is evaluated with its own fee at the current rate, including the marginal cost of the change output, and selection only terminates when a set covers amount plus its own fee.
Fee estimation belongs inside the selection loop, never after it.
If you run on-chain payouts: does your selector price the change output, or discover it later?
https://polycratia.com/c/43729555
- $ · permalink
Every KYC integration I've seen treats the vendor decision as the hard part. It isn't. The hard part is what happens when the vendor says "maybe".
Automated verification gives you three outcomes, not two: pass, fail, and needs a human. That third bucket is where the system actually lives. On a fiat-to-crypto onramp I built, the review queue was a first-class service — not a spreadsheet, not an admin page bolted on at the end.
Because a queue has properties nobody plans for. A user sits in limbo with money already in flight. Two moderators open the same case. Someone approves an applicant whose documents expired while they waited. A retry from the vendor arrives after a human already decided, and now you have two verdicts for one identity.
So I design the moderation path the way I'd design a payment path: explicit states, one owner per case, decisions written as immutable events rather than a status column someone overwrites. The audit trail isn't a compliance checkbox — it's the only way to answer "why is this account open" six months later.
What I'd tell anyone starting: build the human-review flow in the first sprint, not the last. It determines your onboarding latency more than the vendor's SLA does.
If you run KYC in production — what's your actual bottleneck: vendor response time, or the queue behind it?
- $ · 7 min readExactly-once deposits: a broker cannot deduplicate a reorg
A service that credits user balances from on-chain deposits has to tell its consumer about each deposit exactly once. The usual reflex is to push the problem onto the transport: turn on…
- $ · permalink
Hardcoding 18 decimals is a bug that only surfaces in production, and only with real money.
ERC-20 does not fix precision. Each contract reports its own decimals value, and the client is expected to ask. Most tokens answer 18. Several of the ones people actually move — USDT among them — answer 6. Same interface, different arithmetic.
Get that wrong and nothing throws. The transfer encodes cleanly, the transaction succeeds, and the amount is off by twelve orders of magnitude in one direction or the other. On-chain that is final.
So I stopped treating precision as a constant. In erc20-transfers, an open-source project I maintain, decimals is read from the token contract and cached per token per chain — never assumed, never inherited from a config file someone copied between environments.
Conversion happens once, at the edge. Human-readable decimal in, integer base units through the whole system, human-readable out. No float touches the middle.
The part that took longer to accept: the encoder refuses a value the token cannot represent. Ask it to send an amount with more precision than the token has, and it errors instead of truncating. Rounding money is a business decision. A transfer library is the wrong place to make it silently — a rounded-down remainder that no one chose becomes a reconciliation ticket weeks later, when the only evidence left is a hash.
Most integration bugs I have hit in crypto payments are not cryptography. They are unit conversion wearing a serious hat.
I keep the longer write-ups on my blog, if this is the kind of thing you deal with: https://polycratia.com/c/60418fe5
For those running token transfers in production — where do you draw the line on rounding? Reject at the edge, or accept and record the remainder somewhere explicit?
- $ · permalink
Every team I've seen chase exactly-once deposit notifications went looking in the queue. The answer was in the watcher's database.
Here is the shape of the problem. A watcher polls the chain, sees a deposit, publishes a notification. Then it polls again and sees the same deposit. Then it restarts mid-batch and re-scans the block range it already processed. Then a reorg rewrites the block it read. None of that is the queue's business — by the time a message exists, the duplicate has already been created upstream.
So the decision moves earlier. Before publishing anything, the watcher writes a row keyed by the thing that is actually unique on-chain, and lets the database reject the second attempt. Publish only if that write is new. Restart-safe, poll-safe, and it survives a consumer that resubscribes from an older offset.
My take: delivery guarantees move messages, they do not define identity. Identity is yours to persist. Once the watcher owns it, the consumer's queue semantics stop mattering — at-least-once delivery becomes acceptable, because the duplicate never entered the stream in the first place.
I wrote up the full reasoning here: https://polycratia.com/c/09828aa7
If you run a chain watcher: what is your dedup key — transaction hash, hash plus output index, or something you derived yourself?
- $ · 9 min readA deposit is confirmed by the tip, not by first sight
Crediting on-chain deposits fails in a specific way. The watcher sees a transfer, writes a row, and starts incrementing a confirmations column on a timer. Two things are already wrong: the count is…
- $ · permalink
Most KYC integrations fail not at the provider API, but at the moment a human says "actually, look again".
The provider side is the easy part. You send documents, you get a decision, you store it. What breaks is everything downstream of that decision.
Because verification is not a boolean. It is a state that moves — sometimes backwards. A user gets approved, then a document expires. A compliance officer reopens a case after a rescreen. A borderline profile sits in manual review while the user is already trying to move money.
I have built onramps where KYC moderation happened in real time, with a compliance team working a live queue. What made those systems survivable was refusing to collapse verification into a single flag on the user row.
Instead: a case with its own lifecycle, an append-only trail of who decided what and on which evidence, and a separate question the payment path asks — is this user allowed to do this specific action right now. Not "is_verified".
The difference shows up the first time you have to explain to a regulator, or to your own ops team, why a transaction was permitted eight months ago.
If you run KYC in production: what forced you to redesign first — re-verification, or the audit trail?
- $ · permalink
A deposit with six confirmations right now can have two confirmations an hour from now.
That single fact breaks most first attempts at a deposit watcher, because confirmations get stored as a counter and incremented once per poll. They are not a counter. They are a value derived from the current chain tip, and the tip moves — occasionally backwards. Increment it and you end up crediting a balance against a block that no longer exists.
The second half of the problem is the one users actually see. Across polls, restarts and re-scans, the same first inclusion surfaces again and again. First inclusion is an event. Exactly-once notification is the product: one message per deposit, per user, however many times the watcher happens to observe it.
So I keep them as two separate problems with two separate fixes. Recompute depth against the tip on every pass instead of trusting stored state. Key the notification on the deposit itself, never on the observation that produced it — idempotency belongs at the notification boundary, not inside the poll loop.
Where does your watcher break first: reorg handling, or duplicate notifications after a restart?
https://polycratia.com/c/e55a4bac
- $ · 9 min readCoin selection should return a reason, not None
A withdrawal fails at three in the morning and the log line says the coin selector returned None. That one value covers at least four different situations: two of them are answered by funding the…
- $ · permalink
Confirmations are not a counter you increment. They are a value you recompute.
That distinction decides whether a deposit system survives a reorg or silently credits money that no longer exists.
The tempting design is obvious: see the transaction in a block, store confirmations = 1, bump it on every new block, credit the balance when it crosses the threshold. It reads cleanly and it works — right until the chain reorganizes. Then your stored number describes a history that was discarded, and nothing in the data model knows that. The deposit keeps aging toward final on evidence that no longer exists.
In chain-watch, an open-source watcher I work on, confirmations are never stored as a running count. They are derived on every poll: current tip height minus the height of the block that includes the transaction, recomputed against the chain as it is right now. If that block is no longer on the canonical chain, the deposit falls back to unconfirmed on its own. There is no dedicated reorg handler, because there is no stale counter to repair. Reorg survival stops being a feature and becomes a property of how the state is computed.
The part people underestimate is what happens downstream. If a credited deposit later becomes orphaned, you cannot fix it by deleting the credit. In any ledger that has to be auditable, the original posting stays and you write a compensating entry against it, with a reference back to the transaction and the block that vanished. Otherwise you have a balance that changed with no explanation anyone can reconstruct six months later during a dispute.
So the rule I hold to: treat block inclusion as evidence, not as a fact. Evidence gets re-evaluated against the current tip every cycle. Facts get written once — and on-chain, almost nothing qualifies as a fact on first sight.
I write up the longer version of these design notes on my blog, including how the watcher models deposit state: https://polycratia.com/c/e9fcca71
If you run custodial deposits: where does your system store confirmation state — derived from the tip, or as a number you increment?
- $ · permalink
A coin selector that returns None is a bug report with the body deleted.
At 3am that null is the only thing standing between an operator and a stuck withdrawal, and it says nothing actionable. Underneath it are at least three different situations that need three different humans doing three different things.
Not enough total value: someone tops up the hot wallet.
Enough value, but it lives in dust: someone schedules a consolidation, ideally at a quieter fee level.
Enough spendable value, but the input that would cover the shortfall costs more in fee than it contributes: nobody should do anything except wait or change the policy.
Same null, opposite responses.
The second half of this is why selection cannot be a clean one-shot function at all. Fee depends on which inputs you pick. How many inputs you need depends on the fee. Estimate after selecting and you hand back a plan that stops covering itself the moment you add the input that pays for it. Fee estimation belongs inside the selection loop, and when the loop cannot converge, the loop is what knows why.
So: return the reason, not the absence.
How do you type selection failures in your wallet — enum, error subclasses, or a structured diagnostic the support tooling can read?
https://polycratia.com/c/ebeaffa6
- $ · 10 min readA balance is a query, and a hold is not a column
Custodial systems lose money in two ordinary writes: UPDATE accounts SET balance = balance + :amount, and the one next to it that nudges a frozen column up and down while a withdrawal is pending.…
- $ · permalink
A coin selection that returns None tells you nothing. Every support ticket after that starts with a guess.
I hit this building custodial BTC wallets. A withdrawal fails, and someone has to work out which of three unrelated situations they are actually in.
The user genuinely does not have the balance.
The balance exists but sits in dust that costs more to spend than it is worth.
The amount is fine, but the fee at the current rate exceeds the value being sent.
Same None. Three different responses: top up, consolidate, or wait and resize the transfer.
So in utxo-select I made the failure a first-class return value. Insufficient funds, dust-only inputs, and fee exceeding value are distinct outcomes carrying the numbers behind them, not a null. The caller decides what to surface to the user; the library refuses to flatten the distinction in the first place.
The broader lesson from years of payments work is that the empty result is where your operators live. Success paths get designed with care and failure paths get a null, and then the difference between user error and a fragmented wallet has to be reconstructed by hand during an incident.
The code is at github.com/polycratia/utxo-select if you want to see how the outcomes are modelled. I wrote up more of the reasoning behind decisions like this one here: https://polycratia.com/c/e36515ca
Where do you draw the line between a typed failure and a plain null in library code you maintain?
- $ · permalink
A balance column is the most expensive shortcut in a custodial system.
It reads like a fact. It is actually a cache — one you now have to invalidate correctly under concurrency, retries and partial settlement, forever.
I have built custodial wallets for BTC and ETH, a fiat-to-crypto onramp, and per-user virtual account ledgering where money arrived tagged only by requisites and had to be attributed before it could be trusted. The same pattern survived all of them: a balance is derived from paired postings, never stored. You append, you do not update. The number becomes a query, and it stays reconstructible from history instead of from a support ticket.
A hold follows from that. It is not a subtraction from an available column — it is its own object with a lifecycle: reserved, then settled or released, sometimes for less than it reserved. The moment partial settlement is possible, the two-phase object is the only model that stays honest.
And the money type should not belong to the ledger. Amount and currency as a protocol the ledger accepts is what lets card rails, account-to-account transfers and USDT postings land in one journal without the ledger knowing anything about any of them.
Where does your ledger keep the truth today — in a column, or in the postings?
https://polycratia.com/blog/a-balance-is-a-query-and-a-hold-is-not-a-column/
- $ · 9 min readDeclare once, apply to many: scoping VEX decisions without lying
A vulnerability scanner does not remember what you decided yesterday. Every build re-reports the same CVEs against the same dependencies, and someone re-reads the same advisory to reach the same…
- $ · permalink
The most dangerous field in a VEX document is the one filled in by a default.
When you generate VEX at scale, most findings are genuinely undecided. The scanner flags a CVE in a transitive dependency, and nobody has yet traced whether the vulnerable code path is reachable from your entry points. That is not a bug in the process. That is the normal state of a queue that is longer than the day.
The temptation is to make it disappear. Unknown reachability becomes not_affected, the document goes green, and the pipeline stops complaining.
But not_affected is an assertion. It says a human or a tool examined this and concluded the vulnerable function is never called. If nobody did that, the document is now carrying a judgement that does not exist anywhere in your organisation. The consumer downstream cannot tell the difference between analysis and a default value. That is how a compliance artifact turns into a liability.
I built vexdesk (github.com/polycratia/vexdesk) around the opposite rule: an undecided finding is emitted as under_investigation, never quietly resolved. The status is uncomfortable to look at, and that is the point. It puts real work on a real queue instead of hiding it behind a green field.
I have made the same trade before in payment reconciliation. When an incoming payment cannot be matched to an obligation with confidence, you do not attach it to the closest plausible account. You flag it as unmatched and let a human decide. The unmatched pile is ugly, it is visible, and it is honest. Every system I have seen that auto-guessed instead ended up with balances that were silently wrong for months.
The general principle: a system should never manufacture certainty it was not given. "I do not know yet" is a legitimate output, and an artifact that cannot express it will lie by omission.
For anyone generating VEX or SBOM attestations in CI — how do you keep the under_investigation pile from becoming a permanent parking lot?
- $ · permalink
Minting a token that represents a private aircraft is the easy part. Deciding what makes that token wrong is the hard part.
I built NFT generation and lifecycle on Ethereum for high-value physical assets, where the token accumulates signed documents and deal state from listing through closing. The instinct is to treat the token as the source of truth. It isn't. The asset exists in the physical world, the obligations exist in signed paper, and the chain only knows what someone chose to write to it.
So the token drifts. A deal stage advances off-chain, a signature is collected but never anchored, a party walks away — and on-chain state now claims something the world no longer supports. Nothing reverts it, because the chain is append-only and perfectly happy holding a stale claim forever.
What worked: no state transition without a signed document behind it. The document and signature live off-chain in the workflow system; the chain holds the commitment and the ordering. The token becomes a log of attested events, not an assertion of fact.
Most of the engineering in an RWA deal turns out to be e-signature and document lifecycle. The chain is the smaller half.
If you're building RWA flows: what do you anchor on-chain, and what stays in the document system?
- $ · permalink
A balance is not a number you edit. It's a number you derive.
The moment a ledger stores balance as a column and updates it in place, every reservation becomes an UPDATE — a pending payout, an investor commitment waiting on moderation, an on-ramp order not yet confirmed. You get a correct-looking number and no way to answer why it is that number.
So I treat a hold as its own journal object. Written once, with an amount, an owner, and a state. Later it settles into a real posting, or it releases. Nothing in between rewrites history, and available balance becomes derived: posted entries minus active holds.
The case that breaks naive implementations is partial settlement. A hold rarely settles for exactly what was reserved — an authorization captures less, a payout goes out net of fees. That is two facts, not one edit: post the actual amount, release the remainder, both traceable to the same hold.
The honest trade-off is read cost. Deriving a balance from full history is slow, so you need checkpoints — and a checkpoint is a cache, with all the invalidation problems caches have.
I pulled the pattern into a small reference implementation at github.com/polycratia/ledger-core, mostly because I kept re-explaining the same design in reviews.
If you run holds in production: what closes them — the provider callback, or a sweeper you trust more than the callback?
- $ · 7 min readMulti-carrier shipping rates are quotes, not prices
The shipping amount a customer sees at checkout comes from a carrier API call that happened seconds or minutes earlier, computed from package dimensions and an address that can both change before…
- $ · permalink
Everyone building a custodial wallet worries about key storage. The part that actually breaks is the withdrawal queue.
Key management is a solved problem with known answers. Withdrawals are not, because they span two systems that cannot share a transaction. Your ledger debit is reversible. The broadcast is not. Between those two moments sits every incident I have seen in custodial systems since 2018.
A node call times out and you do not know whether the transaction was broadcast. A retry looks safe on the application side and is catastrophic on chain. A user's balance drops before finality and rises again after a reorg. A nonce is reserved by one worker and consumed by another.
The approach that has held up for me: the ledger debit and the intent to send are written in the same transaction, and nothing broadcasts from that path. A separate worker owns broadcast, owns the nonce, and is the only thing allowed to move a withdrawal forward. The chain state is treated as an external fact that gets observed and recorded, never as something the request path can assume. Every state change is append-only, so a stuck withdrawal can be read as a history rather than guessed at.
The useful reframe: a withdrawal is not an action. It is a long-running agreement between your ledger and a chain that does not know your ledger exists.
For those running custody in production — where does your withdrawal path lose certainty first: broadcast ambiguity, nonce contention, or confirmation depth?
- $ · permalink
KYC is not a checkbox at signup. It's a long-lived state machine you have to keep running.
Most systems I've picked up model it as a boolean on the user row: verified, true or false. That holds until the first real case.
A provider returns "pending manual review" while the user is already mid-deposit. A document expires while the account stays open. A re-check months later comes back with a different risk decision. A compliance officer overturns an automated pass.
Each of those is a transition. If your schema stores only the last answer, you lose the reason and the ordering — the exact two things you get asked about later.
When I built a fiat-to-crypto onramp with real-time moderation, the hard part was never the provider integration. It was this: every decision written as an immutable event with its source recorded (provider, internal rule, or human), the user's status derived from that history rather than overwritten in place, and permissions gated per action instead of globally. Deposit, withdraw, and raising a limit are not the same bar.
The payoff shows up on the day a case is disputed. You replay the history instead of reconstructing it from application logs.
If you run verification in production: does your system store the decisions, or only the current verdict?
- $ · 10 min readCustodial ETH withdrawals: the nonce is a database row, not a node call
A custodial wallet service has to turn a user's withdrawal request into exactly one on-chain transaction, while every layer around it — HTTP clients, queues, restarts, impatient users — is…
- $ · permalink
The blockchain part of a custodial wallet is the easy part. The hard part is deciding when a deposit becomes money the user can spend.
Watching addresses and parsing transfers is a solved problem. What is not solved for you is finality. A transaction you saw in the mempool can vanish. A block you credited against can be reorganized away. And the user is already withdrawing.
When I built custodial BTC and ETH wallets and an on-chain payment system, the design that held up was to stop treating "deposit detected" and "balance available" as the same event. Detection creates a pending ledger entry. Confirmation depth promotes it. Only the promotion touches spendable balance, and the promotion is a normal, replayable ledger operation rather than a special case.
That gives you one thing that matters more than speed: a reorg becomes a compensating entry, not an incident. You already have the vocabulary to reverse it, and the history shows what happened instead of a silently corrected number.
The trade-off is user-visible latency, and every product wants to shorten it.
If you run custodial balances: where does your confirmation threshold come from — the asset, the deposit size, or the user's withdrawal behaviour?
- $ · 8 min readThe only safe failure state for an outbound payout is unknown
Sending money out of a system is not the mirror image of taking it in. An inbound payment can be re-parsed, replayed and reconciled at leisure; an outbound payout that leaves twice is a loss…
- $ · 8 min readSplitting a loan repayment across investors without losing a cent
A single borrower repayment often funds a loan held by many investors at once, and the split has to be exact: every minor unit that arrives has to leave the account assigned to somebody. Percentage…
- $ · permalink
Signing a transaction is the easy part of a custodial wallet. Deciding when the money actually left is where the system breaks.
I have built custodial BTC and ETH wallets and on-chain payment flows, and the recurring failure is not cryptography. It is that you now run two ledgers with different notions of finality. Yours is transactional and instant. The chain's is probabilistic, reorganisable, and occasionally silent for hours.
If a user balance is derived from chain state, every confirmation-depth choice becomes a product decision made by an engineer at 2am. If it is derived from your own ledger, you have to explain every gap between what you recorded and what the chain settled.
What I do now: the internal ledger owns the balance, and the chain is treated as an external settlement source that gets reconciled against it — same way I would treat a bank's statement in a fiat payment system. A withdrawal moves through intent, broadcast, and confirmed as separate recorded states, never one boolean. Stuck and replaced transactions then become normal cases with a place to live, not incidents.
The useful reframing: on-chain is not your database. It is a counterparty you reconcile with.
For those running custodial flows — do you let confirmation depth vary by amount, or keep one fixed rule and eat the tail risk?
- $ · permalink
Detecting an incoming crypto deposit is the easy part. Deciding when it is real is what breaks custodial systems.
Your node tells you a transaction exists, then that it landed in a block. Neither fact means a user balance should move. How many confirmations you wait for is a risk decision — how much you are willing to lose against how long a user will tolerate a spinner — but in most systems it ends up as a constant hardcoded by whoever wrote the deposit watcher first, identical for every asset and every amount.
Then a reorg happens, and you find out your ledger has no state for "credited, but still reversible". If the only states are pending and settled, un-crediting becomes a manual adjustment. And if the user already withdrew, the adjustment is just a record of your loss.
How I build it now: the watcher records observations, never balances. A separate crediting policy decides — per asset, per amount — when an observation becomes a ledger entry. Reversals are ordinary ledger entries, not admin surgery.
The chain is the source of truth about transactions. It is never the source of truth about your balances.
Where do you draw the confirmation line — fixed per asset, or scaled with the amount at risk?
- $ · 8 min readTreat LLM Catalog Translation as a Cache, Not a Job
Translating a product catalog with an LLM looks like a batch job: read rows, call the model, write rows back. It stops looking like that the first time a supplier re-uploads the same feed and you…
- $ · 7 min readKYC webhooks arrive out of order, so rank the decisions
An identity verification provider does not hand you a status. It hands you a stream of webhook events, and that stream arrives out of order, duplicated, and occasionally after one of your own…
- $ · 8 min readCrediting On-Chain Deposits Without a Balance Column
A custodial wallet has to answer one question honestly: how much of this user's money is spendable right now. The obvious implementation increments a balance column when a transaction appears in a…
- $ · permalink
A catalog parser that crashes is a good day. The expensive one keeps returning valid data that no longer means what it used to.
I have run retail catalog parsers for a cross-border shopping platform for years, and the pattern repeats. The selector still matches. The price is still a number in range. But the page started showing a member price instead of the retail one, or the unit shifted from a pack to a single item, or a variant field moved and every size now maps to the default. Nothing throws. Downstream, the listing is wrong, the delivery rate is quoted against the wrong weight, and you find out from a customer rather than from a log.
So I stopped treating parsers as code that either works or errors, and started treating extraction as data with an expected shape. Field-level presence rates, value distributions per source, and a diff against the previous crawl. A sudden jump in the share of products with no discount, or a currency that appears where it never did before, is a stronger signal than any exception.
Structural change is easy to detect. Semantic change is the one that costs money.
If you run scrapers at scale: what actually tells you a source changed meaning rather than markup?
- $ · permalink
Most teams build KYC as a gate. The user passes, you flip a boolean, and you move on.
That boolean is the bug.
Verification is not a fact about a person. It is a claim made by a provider, at a specific moment, based on documents that were valid then. Documents expire. Sanctions lists change. Risk scores get revised. A user who cleared verification in March may not be clearable today, and nothing in your system will tell you that, because you stored an answer instead of a decision.
When I build compliance tooling now, I store the whole decision: which provider responded, what they returned, which ruleset version was applied, and when. Approval becomes a dated record, not a permanent property. Re-checks are scheduled, not triggered by someone noticing.
The practical payoff shows up on the moderation side. Your compliance team stops asking "is this user verified?" and starts asking "what did we know, and when did we know it?" — which is the only question that survives an audit.
The part that always causes an argument: what happens to an in-flight transaction when a user's status goes stale mid-flow. Block it, let it settle, or hold and escalate?
How do you handle that one?
- $ · 9 min readMatching Incoming Bank Transfers to Users When the Reference Is Wrong
A bank statement line gives you an amount, a date, a sender, and a free-text reference field, and your system has to decide which user that money belongs to. The reference field is wrong, truncated…
- $ · permalink
The hardest part of a custodial wallet is not key management. It is deciding the moment a deposit becomes real.
Key handling is a solved problem: isolate the signer, keep it off the API path, limit who can trigger it. Tedious, but well understood.
The ambiguity lives on the credit side. A transaction in the mempool is not money. One confirmation on one chain does not mean what it means on another. A reorg can take a deposit back after you have already shown the user a bigger number. Product wants instant credit; the ledger wants finality. Those two pull in opposite directions, and the gap between them is where custodial systems quietly lose money.
What I do: keep pending and confirmed as separate ledger states, and let only confirmed balance be spendable. Every credit is idempotent on the chain identifiers of that specific output, so a node rescan or a replayed block cannot double-credit anyone. Confirmation depth is a policy value per asset, not a constant buried in code. And a deposit that reorgs out gets a compensating entry, never a deletion.
The chain is an input. Your ledger stays the source of truth.
Where do you draw the instant-credit line — a fixed confirmation depth, or tiered by amount?
- $ · permalink
Translating a product catalog with an LLM is not a translation problem. It is a cache invalidation problem.
The model call is the cheap part. The hard part starts the moment the source catalog moves, and retail catalogs move constantly. A title gets a new size suffix, a supplier rewrites a description, an attribute appears that was never there before. Now you have to decide which of your translated fields are stale and which are still fine.
Re-translate everything on every sync and your cost and latency scale with the size of the catalog instead of the size of the change. Re-translate nothing and the storefront quietly drifts away from the source until someone finds it in a support ticket.
What works for me: store each translation keyed on a hash of the exact source fields that fed the prompt, plus the prompt version and the target locale. Change any of those and the entry is invalid. Change none of them and you serve what you already have. The model becomes a fill-on-miss function behind a cache rather than a stage in the pipeline.
The useful side effect is that prompt changes become reviewable. A new prompt version is just a new key, so you can roll it through a slice of the catalog instead of rewriting the whole store in one job.
If you run LLM pipelines over data that keeps changing: what do you key on, and where does that key first betray you?
- $ · permalink
A crypto deposit is not an event. It is an opinion that gets stronger over time.
Most custodial wallet code treats it like an event anyway: watch the chain, see the transaction, credit the balance. That works until the block it landed in stops existing.
What I learned building custodial BTC and ETH wallets is that the interesting design decision is not the confirmation threshold. It is what the user can do with money that is only probably theirs.
So I model a deposit as a state, not a notification. Detected, pending, credited, reversed. Each state has its own rules about what it unlocks: visible in the interface early, spendable late, withdrawable off-platform latest of all. A reorg then becomes a normal transition instead of a support ticket and a manual database edit at midnight.
The part teams skip is the reversal path. Everyone writes the credit logic. Almost nobody writes the code that takes it back and leaves an auditable trail, because it feels like an edge case until the day it is not.
If you run custodial balances: do you let users trade on unconfirmed deposits, and if so, who eats the loss when the chain disagrees?
- $ · permalink
KYC is not a signup step. It is a state that changes over the whole life of the account, and most systems store it as a boolean set once.
That boolean is where things break later. A document expires. A provider re-runs screening and the result flips. A compliance officer reverses an automated approval after manual review. Sanctions and watchlist data changes underneath users who were cleared months ago. Nothing in the signup flow ever notices, because the flag was written once and never asked again.
Building onramps and verification tooling, I stopped storing a verification result and started storing a history: each decision as its own record, with the source (provider or human moderator), the reason, and the period it is valid for. Current status is derived, never overwritten. Manual moderation writes a new decision instead of editing the old one.
Then every money-moving action asks the same question at execution time — is this user cleared right now — instead of trusting a flag set at registration.
The practical payoff is not elegance. It is that when a regulator or a partner asks why a specific withdrawal was allowed on a specific day, you can answer with data instead of a guess.
For those running verification in production: what finally forced you to make KYC status time-aware — an expiry, a reversal, or an audit?
- $ · permalink
Integrating a shipping carrier is not an API mapping problem. It's a state problem you don't control.
Every carrier exposes its own status vocabulary, and the naive move is to map each one onto your own enum and call it done. That works until you run several carriers at once. Then you learn that events arrive out of order, that a shipment can move backwards from out-for-delivery to in-transit, that some carriers repeat a status for days and others go silent for a week, and that delivered is not terminal because a return can reopen the whole thing weeks later.
What I ended up doing on cross-border delivery pipelines: store the raw carrier event exactly as received, with its own timestamp, and treat it as an append-only log. Derive my status from that log rather than overwriting a status column. Order by the carrier's event time, not arrival time. And keep an explicit unknown state instead of forcing every foreign status into a bucket that looks tidy but lies to support and to the customer.
The rule I'd keep: never let an external system write directly into your state field.
For those running multi-carrier shipping — what broke first for you: status mapping, silent gaps, or returns after delivery?
- $ · 6 min readTranslating a product catalog with an LLM: cache keys and guard rails
The first version of an LLM catalog translation pipeline usually works and is still wrong. It re-translates fields nobody touched, and it happily publishes a description where the model rewrote a…
- $ · permalink
The hardest part of a custodial wallet is not key management. It's that a withdrawal is a distributed transaction between your database and a chain that will never roll back for you.
The naive version looks fine in review: open a transaction, debit the user's balance, call the node to broadcast, commit. It survives testing because nothing fails in testing.
In production the node call times out after the transaction is already in the mempool. Now you either roll back and pay out twice, or commit and hope. Neither is an answer.
How I build it instead: the debit and a pending withdrawal record commit together, and nothing touches the chain inside that transaction. A separate worker picks up the pending row, broadcasts, and moves it through a state machine — pending, broadcast, confirmed, failed — keyed by the transaction hash. Broadcasting the same signed payload twice is harmless. Signing a second one is not, so the nonce or UTXO selection has to be owned by one writer.
The reconciler then walks confirmed chain state back against the ledger, and anything that does not match gets flagged rather than auto-corrected.
The rule I keep coming back to: never let an irreversible external effect share a commit boundary with your own state.
For anyone running custodial withdrawals — where does yours get stuck most, stuck broadcasts or the fee bumping?
- $ · permalink
In payments, the retry is not the edge case. It's the normal case.
Every hop between you, the bank, and the card network can time out after the money already moved. The request succeeded; the response got lost. So the client retries, the webhook fires twice, the reconciliation job replays a batch — and if each of those can charge or credit again, you will double-process. Not might. Will.
The mistake I see most is treating idempotency as an API-gateway concern: dedupe the incoming request by a header key and call it done. That catches the naive double-click and nothing else. The dangerous duplicates come from inside your own system — a queue redelivering, a cron rerunning after a crash, two workers grabbing the same row.
So I push idempotency down to where the money actually changes: the ledger write. Every financial effect carries a deterministic key derived from the business event, not the transport. Applying the same key twice is a no-op that returns the original result. The API layer, the queue, and the retrying client can all be as sloppy as reality forces them to be, because the last line of defense is the one place that can't afford to be wrong.
The trade-off is real: those keys have to be stable across code changes, and picking what goes into them is a design decision you live with for years.
Where does duplicate processing bite you first — provider webhooks, your own message queue, or replayed batch jobs?
- $ · permalink
A Python 2 to 3 migration is not a syntax problem. It's a data problem wearing a syntax costume.
The automated tools rewrite prints and imports in an afternoon. What they cannot touch is every place the old code treated bytes and text as interchangeable: file reads, socket payloads, database drivers, anything pickled or cached back when nobody had to decide.
Under 2 that ambiguity got resolved at runtime, usually by accident. Under 3 it becomes a boundary you have to declare explicitly, and the codebase never recorded where those boundaries were.
The failures don't surface in tests. They surface on the one row with an unusual encoding, the legacy record written by a service that no longer exists, the comparison that used to work because 2 would order anything against anything.
How I approach it: map the I/O edges first and pin encoding at each one, then run both versions against production-shaped data and diff the outputs instead of trusting a green suite. Migrate boundary by boundary, not module by module.
Nothing in the language will point at those edges for you. Only real data will.
If you've been through one: where did yours break first — the ORM layer, the caches, or file and CSV handling?
- $ · permalink
The first time a payment integration bit me in production, it wasn't a failed charge. It was a successful one that ran twice.
Every link in a payment chain retries: the client on a dropped connection, the queue on a timeout, the provider's webhook that fires again because your 200 got lost on the way back. None of them know the operation already happened. At-least-once delivery is the default you inherit whether you designed for it or not.
So the job was never "make sure this runs." It was "make sure running it three times leaves the same result as running it once."
The way I approach it now: an idempotency key minted at the true origin of the intent, not regenerated per retry. A persisted record of that key and its outcome before any side effect touches money. Second attempt looks up the key, sees the settled result, and returns it instead of moving funds again. The dedup has to live in the same transactional boundary as the write it guards, or you've just moved the race one layer down.
The trap is scoping the key too narrowly and treating a legitimate second payment as a duplicate, or too broadly and swallowing one the user actually meant to send twice.
Where does your idempotency boundary sit — at the API edge, the queue consumer, or the ledger write?
- $ · permalink
A catalog scraper almost never fails loudly. It fails silently, and that's what makes it dangerous.
When you pull product data from retail sources you build for the happy path: fetch the page, match the selectors, extract price, stock, title, images. It works, so you move on. Then the source ships a redesign, or splits one field into two, or starts A/B testing a new layout on half its traffic. Your scraper keeps returning 200. It keeps writing rows. Nothing throws. But the price is now the crossed-out price, or stock reads "in stock" from a cached banner, and your catalog is quietly wrong for days before a human notices a listing that makes no sense.
After years running parsers across many retail sources, I stopped treating extraction as the hard part. Fetching is easy. The real work is trusting what you fetched.
So I build the parser to distrust itself: shape checks on every extraction, ranges each field must fall in, and a diff against the last known-good snapshot per source. A sudden jump in null rates or a price that moved an order of magnitude doesn't get written — it gets quarantined and flagged. I'd rather serve stale data I know is stale than fresh data that's silently corrupt.
For those running scrapers at scale: what's your first line of defense against a source that changes shape without telling you?
- $ · permalink
Minting an NFT for a physical asset is a weekend job. Making it survive a real deal is the part nobody warns you about.
I built NFT lifecycles on Ethereum for high-value physical assets — private aircraft, where a single transaction runs from listing to closing over weeks. The token isn't a picture. It's a container that has to accumulate signed documents, escrow state, compliance checks, and multi-party approvals, all while staying consistent with an off-chain reality that keeps moving.
The hard questions are never on-chain. What is canonical when a document is signed off-chain but the token hasn't caught up? Who can advance the deal state, and what stops a stale client from advancing it twice? Where does the escrow release actually gate?
My rule: the chain records agreement, not truth. Truth lives in a reconciled backend that treats every on-chain event as a claim to verify, not a fact to trust. The token references state; it doesn't own it.
If you've put real assets on-chain: where did your on-chain and off-chain state first drift apart, and how did you catch it?