The endpoint that takes 3 seconds is usually not the endpoint that is slow.
Latency gets attributed per route, but the resources behind a route are shared: a connection pool, a worker pool, a provider's per-account rate limit. When one of them saturates, the cost lands on whatever request arrives next, not on the request that caused it.
The mechanics are mundane. A handler holds a database connection while it waits on something that is not the database: a carrier rate call, a provider confirmation, a file upload. Query time stays flat. The pool does not. Every other request now spends most of its life queued before it executes, and your trace shows three seconds inside a handler that did almost nothing.
The signature is recognizable. The p50 looks fine, the tail is awful, and the guilty route keeps moving between deploys, which is the tell that none of them is guilty.
So I instrument time-to-acquire separately from time-to-execute: pool wait, worker wait, lock wait, each as its own number. Once those are visible the fix is architectural instead of clever: do not hold a connection across a network call, and give external calls their own bounded pool so they can only starve themselves.
An endpoint's latency is not a property of its code. It is a property of what it stands in line behind.
I would like to hear which shared resource caught you out last: connections, workers, or someone else's rate limit...