An endpoint that answers in almost exactly three seconds is rarely slow code. That is a retry budget being spent in full.
Slow code gives you a messy distribution. Work scales with input: more rows, more time. The histogram smears.
Retries do the opposite. Latency goes quantized. You get a cluster near zero, another near one second, another near two, another near three, and almost nothing in between. That shape is not computation. That is a timeout firing, a backoff sleeping, and an attempt starting again, stacked inside a single request the caller believes is one call.
In payment and carrier-rate integrations I have watched this hide for a long time, because each layer on its own looks reasonable. The HTTP client ships with retries enabled by default. The provider SDK adds its own. The service wrapping both has a policy too. Nobody configured a three-second endpoint: three independent one-second decisions composed into one.
The tell is in the shape before it is in the profiler. Stop reading the average, plot the raw distribution, and ask what round number the mass is sitting on. Round numbers come from configuration, not from work.
Two things I now treat as non-negotiable. Retries belong in exactly one layer, and every other layer gets them switched off explicitly. And the total time budget for a request is set at the edge, then passed down as a deadline, so an inner call cannot spend more than the caller is still willing to wait for.
The second one matters more than it sounds. Without a propagated deadline, a retry is just a request that already lost its reader and is still burning a connection from the pool. That is how one slow upstream turns into a saturated service.
I wrote up more of how I approach production systems like this on my blog: https://polycratia.com/c/a6c5253f
The last time you chased a latency number that looked fixed, something was spending it: a retry, a lock, a queue...