The 3 seconds weren't the work. They were a poll interval.
The giveaway was the shape of the latency chart, not its height. Response times didn't spread out the way real work does: they clustered on round numbers. One second. Two. Three. Compute doesn't quantize. Waiting does.
The endpoint handed a job to a worker, then blocked on the result backend, asking "done yet?" on a fixed tick. The job itself finished in a fraction of the time the caller spent waiting. The response just sat there until the next tick noticed it had finished. And if the first poll fires before the task even gets picked up off the queue, you've bought a full extra interval for nothing.
That is why standard instrumentation finds nothing. The handler is fast. The worker is fast. The queue sits near empty. Every component reports healthy, because nobody measures the gap between "result written" and "result observed".
Two rules I apply now.
Anything clustering on a round number in a latency distribution I treat as a wait until proven otherwise. Round numbers come from configuration, not from physics.
And synchronous request/response over a task queue is a design smell. Either do the work inline in the handler and own the latency honestly, or return a job id and let the client subscribe to completion. Blocking an HTTP worker on a poll loop pays for concurrency twice and hands you latency you did not choose.
I unpack more of these failure modes in longer form on the blog: https://polycratia.com/c/b668cb9e
I'd like to hear about the most expensive wait you've found that no dashboard was measuring...