The slow endpoint had a 9 ms database query. I checked it three times because I did not believe the number.
This was a cross-border shopping platform. A product page endpoint took seconds, and the first assumption on the call was "the catalog table needs an index". It did not. Postgres was answering fast and the query plan was boring.
What was actually happening: that one request fanned out to things that were not the database. Delivery rate lookups across multiple carriers, done one after another. A translated-copy call for the localized catalog text. A currency figure. Each one was fine alone: a couple hundred milliseconds, sometimes more when a carrier was having a bad afternoon. Run serially, inside a request a user was waiting on, they added up into a page that felt broken.
The cost was not just the latency. My availability became the worst availability of everyone I called. One slow carrier API and the whole product page degraded, which is a strange thing to explain to a business owner whose own servers are healthy.
What I do differently now: I treat any third-party call inside a user-facing request as something that has to justify itself. Delivery rates get fetched asynchronously and cached per route and weight bracket, with a last-known value served while a refresh happens behind the request. Translated catalog copy comes out of the LLM pipeline ahead of time and gets stored, not generated on read. Anything genuinely live gets a timeout short enough that it can fail without taking the page with it, plus a defined fallback (a stale rate with a label beats a spinner).
The useful habit is measuring where the wall-clock time goes, not where you expect it to go. Count the outbound calls per request before you touch an index. Most "slow API" tickets I have picked up were fast code waiting politely in line for someone else's network.
If you run integration-heavy backends, I would like to hear what finally forced you to move third-party calls out of the request path...