The endpoint took three seconds and came back 200 OK. The 200 was the part that bothered me.
A KYC flow: submit a document, get a verification decision back. Latency had crept up over a week. Nothing on the error dashboard, no alerts, nothing loud enough to page anyone. Just slow.
The handler was fine. The queries were fine.
What I ended up reading was the client wrapper around the verification provider. Inside it sat a retry decorator added years earlier: three attempts, a sleep between them, and a swallow on anything that looked transient.
The provider had been failing calls and succeeding on retry for days. The service absorbed all of it. The only trace left anywhere was wall clock time.
Two costs. I burned a day tuning the layer that wasn't broken, because I read query plans before I read attempt counts. And the provider's submit endpoint wasn't idempotent, so every slow success had been creating duplicate submissions on their side, which turned into manual cleanup with their compliance people.
Now I log attempt number and per-attempt duration on every outbound call, and I alert on retry rate instead of latency. Retries wrap only the calls I know are safe to repeat. Everything else fails loudly and early.
A retried call is a failure you decided to hide from yourself. Latency is just where it leaks out.
I write up more of these production failure modes here: https://polycratia.com/c/720dea45
Somewhere in your system there is probably a slow path that turns out to be a retry nobody remembers adding...