By hand the endpoint was fast every time I called it. Under real traffic it crawled, and the database sat there bored the whole time.
I checked the obvious things first. Queries came back in single-digit milliseconds. The handler barely did anything: read a record, call a payment provider, write a status. Adding workers moved the number a little, then stopped helping.
What finally pointed me at the right layer was the shape of the latency. It got worse as concurrency went up, and it got worse for requests that had nothing to do with the slow path. That is the fingerprint of a queue, not of slow work.
The service was async. One call in that path used the provider's synchronous HTTP client, pulled in through an SDK I had not looked inside. Every request that reached it parked the event loop until the provider answered. So one slow upstream response froze every other in-flight request on that process, including health-adjacent reads that touched only Postgres.
Most of a day went to profiling the database and tuning the connection pool. Both innocent.
Three things changed in how I work after that.
A latency number measured at concurrency of one does not count for me anymore. Event-loop lag sits next to request duration as a first-class metric, because it is the only one that shows this failure. And I audit transitive dependencies for synchronous I/O: a sync client in an async service runs in a thread pool or it does not ship.
I write up the longer versions of these on my blog: https://polycratia.com/c/5b7df874
If you run async Python in production, I'd like to hear what you use to catch loop blocking before your users do...