The database dashboard was green while the endpoint timed out. Same service, same minute.
A payment status callback came in. The handler opened a transaction and then, inside it, called the provider back to confirm the status before writing anything. Code that looks fine. The query itself took milliseconds.
The transaction stayed open for the whole round trip to the provider, though. Under load every worker sat on a database connection while waiting on someone else's network. The pool emptied. Unrelated endpoints that needed one tiny read queued up for a connection that would not come free in time.
My latency charts showed query time, not time-to-acquire-a-connection. So the slow part was invisible exactly where I was looking.
The bill: callbacks timed out at my edge, the provider retried them, and an evening went into reconciling payments that had settled fine.
Now transaction scope is a hard boundary for me. No provider call, no file upload, no queue publish inside a transaction. Talk to the outside world first, then write in one short transaction, and make that write idempotent so repeating the sequence is safe.
And I graph connection wait time next to query time. A fast database with no free connections is still a slow API.
Worth checking where your transactions touch the network right now: provider calls, object storage uploads, notification fan-out...