The slowest endpoint in that service did the least work: a small config read that touched no database at all.
Everything I looked at said the database was healthy. Queries in single-digit milliseconds, no lock waits, connection pool half empty. So I added workers, the easy move. Latency got better for about an hour, then came back exactly where it started.
The service was async. One path through it was not.
A provider SDK sitting in the payment flow made its HTTP calls synchronously, and that call lived inside an async handler. While it waited on the remote response, the event loop in that worker did nothing else: no other request in that process moved. The database looked fast because it was idle, and the requests that needed it never got far enough to ask.
That is why the cheapest endpoint was the clearest symptom. It had no work of its own to hide behind, so what it measured was time spent waiting to be scheduled.
The fix was small: move the blocking client onto a bounded thread executor. The habit change was the real outcome. I now track event loop lag as a first-class metric right next to request latency, and I treat every third-party SDK as synchronous until I have read its client code instead of its README.
Per-request latency tells you how long one request took. It tells you nothing about whether the process could do anything else during that time. Those are different questions, and only the second one explains why unrelated endpoints degrade together.
I write these up in longer form here: https://polycratia.com/c/4e097522
If you run async services in production, I'd want to know what exposed the blocking call for you: loop lag, or a latency graph that stopped making sense...