Locally the endpoint was fast, and in production the median looked fine. Then a slice of requests hit a wall at almost exactly three seconds, every time.
That shape cost me two days. A flat ceiling, not a spread, and I spent those two days looking in the wrong place. I profiled the handler. Checked the queries. Timed the serializer. All of it came back fine.
The three seconds was a retry. Not mine.
The HTTP client wrapping a payment provider integration shipped with a default retry policy: on a connection-level error, try again, twice, with a short sleep between attempts. The provider was failing on a small share of calls, and the library absorbed that quietly. One slow request is what my metrics saw. What actually happened was three outbound calls and two swallowed errors.
The latency was the cheap part of that bug.
Those retries were going out on a call path that was not idempotent. A duplicate charge attempt is a much worse problem than a slow endpoint, and nothing in the logs would have told me it was happening: from the application's point of view, one call went out and one result came back.
So two things changed in how I work.
I read the defaults of every client library that touches money or crosses a network boundary, retries and timeouts and connection reuse included, and I set them explicitly instead of inheriting them. A default I did not choose is still my behaviour in production.
And I log attempt counts, not just request duration. Duration tells you something was slow. Attempt count tells you why, and it is the only signal that makes a swallowed error visible at all.
The heuristic I kept: a timing anomaly with a flat ceiling is almost never computation. Computation varies. Waiting and retrying do not, they land on round numbers, because someone picked that number for you.
I write up more of these production post-mortems in longer form here: https://polycratia.com/c/66ff2bb6
If you have found one of these in your own stack, I'd be curious which exposed it first for you: attempt-level logging, or a duplicate showing up downstream...