The endpoint everyone complained about was doing 40 milliseconds of work.
I checked the handler timing first, then the queries. Both looked fine. Whatever was slow happened before my code ever ran.
A payment status endpoint in the same service was calling a provider synchronously. On its own that call looked harmless, and then you count workers. Each request parked one for the whole duration, and there were only so many of them. Once they were all parked, every other request sat in the accept queue waiting for one to free up. Catalog reads, health checks, login: all of them suddenly around three seconds, and not one of them was slow.
That is what cost me the day. The symptom showed up on endpoints that had nothing to do with the cause, so I went hunting for slow code in places where there wasn't any. The dashboards reported service time and never queue time.
What I do differently now: any outbound call on the request path gets a timeout that is a product decision instead of a library default, and anything that can't honour it moves to a queue and gets polled. I also measure time-in-queue as its own metric, separate from handler time. Without that split, a saturated worker pool looks exactly like a slow database.
Latency you can't attribute to a line of code is probably contention, not computation.
If you run a synchronous API in front of a payment provider, the thing I'd want to know is your isolation at the worker level: separate pools per dependency, or one pool and hope...