A compliance reviewer told me the moderation queue took about three seconds to load. So I instrumented the handler. The handler was fast: its own work ran in tens of milliseconds. The rest of the time went by before my code ran at all.
The request was sitting in line, waiting for a free worker.
Somewhere else in the same service, a different endpoint called an external verification provider synchronously, inside the request cycle. When that provider slowed down, those requests parked on workers and held them. Every other endpoint in the process inherited the wait. Nothing in the moderation code was wrong, and no amount of query tuning there would have fixed it.
Two things changed after that.
I measure queue wait as a number separate from handler duration. Service time and latency are not the same metric, and reporting only the second one hides exactly this failure mode.
And outbound calls to third parties come off the request path, into a task queue with its own timeouts and its own workers, or at minimum they get isolated so they can only starve themselves.
The uncomfortable part: this presents as a performance problem and is actually a blast-radius problem. One dependency's bad afternoon became every user's bad afternoon.
When your p95 climbs, I'd want metrics that tell you whether requests were working or waiting...