polycratia

The database dashboard was green. The slowest query on that page was trivial. The endpoint timed out anyway, and so did endpoints that touched nothing related to it.

It was a verification flow. One request opened a transaction, wrote the applicant record, called the verification provider over HTTP, wrote the result back, committed. Atomic and easy to reason about, and that is why everything fell over.

The provider got slow that day. Not down, just slow. Every in-flight request sat there holding a database connection it wasn't using. The pool drained. After that the latency wasn't in the queries: it was in waiting for a connection to run them. Connection checkout wait doesn't show up in query statistics, so the database kept honestly reporting that it was fine. It was fine. It just had nothing left to hand out.

What I changed, and still do: no network call inside an open transaction. I commit a pending row, call the provider outside any transaction, then open a second short transaction to record the outcome, with an idempotency key, because otherwise a retry can create a second applicant.

The other change was instrumentation. I measure connection checkout wait separately from query time. A fast database with no free connections looks identical to a healthy one from the inside, and nothing like it from the caller's side. I'd go look at where your pool gets held longest: external calls, or something you haven't looked at yet...

react

$ new-project --brief

or email hey@polycratia.com