An endpoint that sits there for 3 seconds is usually not waiting on data. It is deciding, row by row, whether you are allowed to see any of it.
The mechanics are dull, and that is why they survive review. The handler loads a page of records, then asks an authorizer about each one: does this user own it, is their role scoped to this merchant, is the object in a state they may read. Each check costs a few milliseconds. Each one may touch Redis or a second table. Multiply that by a couple hundred rows and you have an N+1 inside the access layer instead of the serializer, and it hides well, because no single span is slow. The database looks healthy. The query log looks healthy. The time is scattered across thousands of operations too small to notice one at a time.
Pagination makes it worse. If filtering happens after the fetch, you read and authorize more rows than you return just to fill a page, so the invisible work scales with how much the user is not allowed to see.
When I redesigned the access model on a payment platform, I did not make the checks faster. I moved the decision into the query: resolve the caller's scope once per request into concrete predicates, push them into the WHERE clause, and let the database do the filtering it is built for. Per-row callbacks collapse into a set membership test.
The metric worth adding: rows examined per row returned. Once that ratio drifts above 1, your latency is a permissions problem wearing a performance costume.
I write up more of these debugging patterns on the blog: https://polycratia.com/c/fcba111e
So it is worth knowing which of the two your authorization runs in: the query, or a loop after it...