The checkout page felt slow and the database dashboard was clean. Every query behind the delivery-rates endpoint came back fast, and not one of them showed up as a problem anywhere.
So I did the obvious thing first: I looked at giving the database more room. It changed nothing, because the time was never in the queries. It sat in the gaps between them.
That endpoint asked several carriers for shipping rates. Each call built a fresh HTTP client, so a DNS lookup, a TCP handshake and a TLS handshake all had to finish before a single byte of rate data moved. Repeated per carrier. Repeated per request. The database was answering in milliseconds while the handler spent most of its life negotiating connections on someone else's network, one carrier at a time.
What it cost was not just latency. It was a week of looking in the wrong place, because the one dashboard everyone trusted was green and technically correct.
What I do differently now:
One long-lived client per process, with keep-alive, created at startup instead of inside the handler.
Independent outbound calls run concurrently, each with its own timeout and its own fallback, so one slow carrier degrades one row and not the whole page.
And I read an endpoint as a timeline of spans, not as a sum of query times. A sum hides dead air. A timeline shows it right away: long flat stretches where the service is waiting on a third party, or on the cost of reaching one.
Database dashboards are honest about what they measure. They just don't measure the part of the request where your service is a client of someone else, and in payments, delivery and KYC flows that part is usually most of the request.
The last time you chased an API latency problem that was not the database, the time probably turned out to be sitting somewhere like that...