The slow endpoint lived in a KYC flow, and it took almost exactly three seconds. Not 2.6, not 3.4. Three, again and again.
Round numbers are rarely slow code. Slow code is noisy.
The traces showed the handler idle for most of that window. The provider call sat on the critical path, and inside the client library there was a retry policy nobody had configured: connection error, sleep a second, try again, sleep longer, try again. The last attempt usually went through, so every dashboard reported a 200 and a latency number. Nothing anywhere was labeled a failure.
What that cost: the provider was dropping connections intermittently, and the system kept that behind a latency chart instead of surfacing it. Worse, some of the retried requests did land. Verification records got created twice, and a compliance operator had to work out by hand which one was real.
What I do differently now: I count retries, not just time them. Every outbound call to a money or identity provider reports attempt count and final outcome, and more than one attempt is a logged event instead of a quiet success. Anything retried that writes state carries an idempotency key from the first try.
A duration metric tells you how long you waited. It doesn't tell you how many times you asked.
If you've chased a flat plateau in a latency chart, mine turned out to be retry backoff. Yours could be a lock, or a connection pool waiting its turn...