A checkout page started timing out, and I spent the first hour staring at the database.
Every query in the trace came back fast. Indexes were fine. The connection pool sat nowhere near saturated. The slow part wasn't in Postgres at all: it was the shipping step.
That platform quotes delivery across several carriers, and the handler asked each carrier for rates, one after another, inside the request. Nobody noticed for a long time, because the calls were quick and their sum stayed under the threshold where a page still feels alive. Then one carrier's API started answering slowly. Not failing. Just slowly.
Failures you catch immediately. Slow success looks like health.
What it cost wasn't a broken checkout, it was worse: a checkout that worked, took far too long, and pushed people into refreshing and re-submitting the same order. Duplicate submissions, support traffic, and a long argument about database performance that had nothing to do with the database.
What I changed, and still do:
I measure time spent inside my own process separately from time spent waiting on someone else's, in the same trace, as two different numbers. If those are one number, you will misdiagnose every incident like this one.
Every outbound call in a request path gets its own timeout, set well below what the client will tolerate.
Fan-outs go concurrent, and the response degrades to partial results instead of waiting around for the slowest participant.
Anything not required to answer the request leaves the request path entirely: quotes that can be pre-warmed, label generation, notifications. None of that belongs between the user and their response.
A fast database only tells you your database is fast. The request is as slow as the slowest thing your handler is willing to wait for, and most handlers will wait forever unless you tell them otherwise.
I write up more of these production post-mortems here: https://polycratia.com/c/9910a389
Next time you profile your slowest endpoint, look at how much of that time was your own code actually running. Probably less than you'd guess...