The complaints were about an endpoint that did almost nothing: read a document's state, return it.
The query behind it ran in well under a millisecond. No loops, no third-party call, no serialization of anything large. And a few times an hour it took seconds.
I spent longer than I want to admit inside that handler. Added timing around the query. Added timing around the response build. Everything I measured came back fast, and the request was still slow.
The cause was a different route on the same service. It verified signatures on signed documents, a CPU-bound cryptographic check written as an ordinary synchronous call inside an async handler. While that check ran, the event loop did not advance. Nothing else on that worker progressed. Every unrelated request sitting there kept piling up latency that belonged entirely to someone else's code path.
That's what made it so expensive to find. The slow endpoint wasn't slow. It was a bystander. And because the delay got spread across everything in flight, it looked like infrastructure noise rather than one bad call.
Two things changed in how I build async services after that.
First, any CPU-bound work or sync-only client library goes to a thread pool or a queue worker by default, not when it starts hurting, from the first commit. Verification, PDF work, image handling, old SDKs with no async support: all of it leaves the loop.
Second, I instrument event-loop lag as a first-class metric, right next to request duration. If loop lag spikes, no amount of query tuning will help, and knowing that immediately saves days.
The mental shift I found useful: in an async service, latency isn't a property of an endpoint. It's a shared resource, and one blocking call spends everyone's share.
I write up more production failures like this one on my blog: https://polycratia.com/c/d437c2c1
If you run async services in production, are you measuring loop lag, or only request duration?