Slow APIs stay unexplained mostly because the database is fast.
Every query comes back in single-digit milliseconds, nothing trips the slow-query threshold, the data layer dashboard is green, and the endpoint is still visibly slow.
Here latency is a count, not a cost: the number of sequential round trips a request makes before it can answer.
The mechanics are mundane. A list endpoint loads its rows and then enriches each one: a lookup per item, a permission check per item, a counter per item. Each call is cheap, and each call waits for the one before it. Storage is fast, so no single hop ever looks wrong, and every tool in the stack measures duration per query instead of depth per request. The request pays for depth; the monitoring reports duration. That mismatch is the whole blind spot.
What I do about it: I treat round trips per request as a first-class number, the way you'd treat p95. Log the count per endpoint, not just the time. Then watch one property, whether that count stays constant or grows with the size of the result set. A handler whose query count scales with how many items it returns is not a performance bug you optimise away. It is a shape you have to change: one batched read instead of per-item reads, or enrichment moved out of the hot path, or the value precomputed at write time rather than assembled at read time.
The uncomfortable part: fast storage buys you room to be wasteful, and that room is exactly where the latency piles up. Teams that migrate to faster hardware and see nothing improve usually land here. They made each hop cheaper without making the request shallower.
I wrote the longer version of this, including how I split request timing so the trips stop hiding behind each other, here: https://polycratia.com/c/858b526b
If you've chased this before, I'd be curious: query count per request constant, or quietly scaling with page size...