The endpoint everyone complained about did almost nothing. It read one row and returned a small object.
My traces said 41 milliseconds. The clients said three seconds.
I spent an embarrassing amount of time on the wrong half of that request. I added an index. I cached the read, and I denormalized a field so the handler would stop joining. The numbers in my dashboard got prettier. Nobody outside noticed any difference.
The problem was my timer: it started where the handler started. Everything before that was invisible to me.
So I moved the measurement outward: one timer at the proxy, one as the outermost middleware, and named segments for each layer in between. The picture inverted right away. Almost all of the time went before my code ran, inside the auth layer.
Every request carried a bearer token, and the middleware resolved that token into roles and permissions by calling the OAuth service. That service was not slow. But the HTTP client got constructed per request, so each call paid name resolution and a fresh TLS handshake before it asked its question. And the resolution happened twice, once in the global middleware and once again inside the route's permission check, because both had been written as if they were the only one.
So the cheapest endpoint in the service was carrying the most expensive preamble, and the health check was paying for it too.
Two things changed permanently after that.
First, introspection results are cached with a short TTL and invalidated explicitly when permissions change, instead of being re-derived on every call. Permissions change rarely: requests arrive constantly.
Second, and more useful: the first thing I instrument in a new service is not the query. It is the span from accepted connection to first byte out, broken into named segments: auth, deserialization, handler, serialization. A layer that is not named in the trace is a place latency can hide, and in my experience that is exactly where it goes...
If an optimization improves your dashboard but not your caller's clock, probably you are measuring the wrong interval.
I write up these breakdowns in more depth, with the reasoning and the trade-offs, here: https://polycratia.com/c/de71581b
If you've chased a slow endpoint recently, it is worth checking how much of your request timeline actually sits inside a named span, and how much is just the gap between the proxy log and the handler.