Six seconds on a verification endpoint. The query inside it took forty milliseconds, and nothing in the logs was red.
I had instrumented the handler and the database, and both looked healthy. The time was going into one outbound call to a KYC provider, and inside that call the client library was retrying. Two failed attempts, backoff between them, the third one succeeded. From my side it looked like a single successful request with a strange duration. No error, no alert, nothing to grep for.
The latency itself was cheap. What it cost me showed up elsewhere: users staring at a spinner resubmitted, and that created duplicate verification sessions, duplicate records landing in the moderation queue and duplicate checks against the provider, two sessions for the same person that a compliance operator then had to reconcile by hand. A latency problem had quietly become a data problem.
So now anything that leaves the process gets measured at the attempt level rather than the call level: attempts per request, duration per attempt, plus a timeout I choose instead of whatever the SDK defaults to. If the call can tolerate waiting, it leaves the request path entirely and becomes a job with a status the client polls.
A fast database tells you almost nothing about an endpoint that spends its time in a place you never counted.
I write up more production post-mortems like this one here: https://polycratia.com/c/bc2c664f
One thing I'm curious about: do you log retry attempts as separate events, or does your tracing collapse them into a single span?