Peak traffic made a category page on a cross-border catalog time out, and the database dashboard stayed flat and green the whole time. Every query in the slow log was fast.
The cache was the problem: specifically, the moment it wasn't there.
Those payloads were expensive to build: catalog rows, carrier rate lookups, currency conversion, serialization. All of it cached, so in steady state the database barely saw the traffic. That is why it looked fast.
The keys had all been warmed by the same job, though, so they expired as a cohort. The instant a popular one went cold, every concurrent request for that category missed, and all of them started the same rebuild. Hundreds of identical computations racing each other, each individually fast, collectively enough to exhaust workers and connections. Endpoints with nothing to do with the catalog queued behind them.
Per-statement timing measures work. A stampede is duplicated work. Nothing in query metrics describes it.
What I do differently now: I jitter every TTL so a warm set doesn't expire together, I put a single-flight lock around recomputation so exactly one request rebuilds while the rest wait on its result, and I serve the stale value while the refresh runs behind it. Stale by a few seconds beats a timeout almost every time.
The uncomfortable part is that a cache makes a system look faster than it is. You aren't measuring your database under load, you are measuring it with most of the load removed. Capacity you never tested is capacity you don't have.
I write up more of these, the failures where the metric was fine and the user wasn't, here: https://polycratia.com/c/39b1693a
If you run read-heavy catalogs, you either guard against the simultaneous miss or you probably just haven't hit the moment where it matters yet...