The trace said 2.9 seconds. The database said 30 milliseconds. Between them sat one span, called "get token".
An internal payments endpoint had been parked at roughly three seconds for weeks. Nobody suspected auth. There was a token cache, I had written it, and it sat right there in the code.
It just never hit.
The cache key got assembled from the request context, and that context carried a value unique to every call. So each request minted a fresh access token from the OAuth service, and that service serialized mints. Under load, the queue in front of that one operation was the whole three seconds.
Latency was the cheap part of the bill. A token per request meant the credential store grew with traffic, revocation lists stopped meaning anything, and a short wobble in the OAuth service failed every request instead of the small share that genuinely needed a new credential. The slowness was just the symptom loud enough to notice.
Two changes stuck.
Hit rate ships with the cache now. Not inferred from the existence of a cache: emitted, graphed, alertable. A cache with no hit-rate number is an extra code path with optimistic naming.
And refresh is single-flight: many callers wait, one mint happens.
I write these up in longer form on the blog, if this failure mode looks familiar: https://polycratia.com/c/28cdd2f2
If you have a token cache in front of an internal service, you probably know the code is there. The hit rate today is the part I'd go look up...