An API that answers in three seconds is usually not doing three seconds of work. It is waiting on its own retry policy.
The mechanics are dull. A handler makes one outbound call: a verification provider, a card processor, a bank endpoint. That call has a timeout, and around it sits a retry with a couple of attempts. On the happy path you notice nothing. When the dependency degrades, every attempt pays the full timeout, and the handler quietly becomes the sum of them. The slow response is the system doing what it was configured to do.
What makes this hard to see is where the time lives. The dependency's tail becomes your median. Your own signals look fine: CPU flat, no slow queries, handler code untouched since last quarter. The latency is spent waiting, and waiting does not show up in the places engineers habitually look.
Two changes have fixed this for me in payment and KYC integrations.
A deadline instead of timeouts. The request gets one budget at the edge, and every downstream call inherits whatever is left of it. An attempt that cannot finish inside the remaining budget does not get made at all.
Retries off the request path. With idempotency keys on the outbound call, the second attempt belongs to a worker and not to the user's open connection. The caller gets a determinate answer now, and the system keeps working behind it.
The general shape: a retry inside a synchronous handler turns an availability feature into a latency bug, and it does that silently.
I write up more of how I design these request paths on my blog: https://polycratia.com/c/4c6f9622
I'd go find out who owns your retry budget: the client, the handler, or nobody in particular...