Three seconds of latency is rarely three seconds of work. Usually it's the same work done three times.
Retry logic hides in layers. A provider SDK retries on connect timeout. The HTTP client wrapping it adds a retry of its own. The task queue retries the job. None of them share a deadline, so a one-second timeout with two attempts and a little backoff puts you at three seconds, while the trace shows a single outbound span, because the retries happen below the instrumentation boundary. The endpoint doesn't look busy. It looks patient.
The expensive part is what the handler holds while it waits. If the external call sits inside an open database transaction, every retry extends lock duration and connection-pool occupancy, and the slowness leaks into endpoints that never touch that provider at all.
How I approach it: exactly one retry layer, picked deliberately, and a request deadline propagated downward so an inner attempt can't outlive the outer budget. External calls move outside the transaction. And any retried write needs an idempotency key, or you've traded latency for duplicate side effects.
Measure attempts, not just durations. A p99 that refuses to explain itself is probably a retry count nobody is logging.
So I'd want to know where your timeout budget actually gets enforced: in the client, at the gateway, or nowhere...