3.0 seconds. Not 2.8, not 3.4. The endpoint came back at three, run after run.
That flatness was the whole clue, and I spent a day ignoring it.
It was a KYC status check on a fiat-to-crypto onramp. I did what you do: added spans, profiled the handler, went looking for the expensive query. The traces summed to roughly 80 milliseconds of real work (one indexed read and a status mapping). Everything else sat in a gap that no span covered.
The missing time was not in my code. It lived inside the verification provider's client library, which retried connection failures on its own: a short connect timeout, a couple of attempts, quiet backoff in between. One of the addresses behind the provider's hostname was not routable from my network. Every request burned the full retry budget before it happened to land on the one that worked.
So the handler was not slow. It was waiting on a clock someone else had set, in a layer I had not configured.
What it cost was not latency. Support started reporting verification as "hanging". Users re-submitted their documents. Each resubmission opened a new case for a human moderator to review. A three-second gap in a dependency turned into a queue of duplicate manual work.
Two things changed in how I build after that.
First: constant latency is a timeout until proven otherwise. Real work has variance, since queries, serialization and payload size all wobble with the data. Waiting does not wobble. A number that stays put is a configured limit, and the number itself usually tells you which one.
Second: I do not let a dependency own its retry policy quietly. Connect timeout, attempt count and backoff get set explicitly at the call site, and every attempt gets logged as its own event. Retries that live only inside a library are invisible in a dashboard: they make p50 look mysteriously bad and teach you nothing about why.
When a client library retries on your behalf, I think it is worth knowing whether those attempts show up anywhere you can actually see them...