That endpoint was fine for almost everyone. It was slow only for the accounts carrying the most data.
A 3-second response usually gets read as a code problem: profile the handler, add an index, move on.
But the median looked fine. So did the 90th percentile. The slowness sat in a thin tail, and everyone in that tail had one thing in common: unusually large related data. On a lending deal that meant many investors sharing one loan, each with a position to resolve. In a catalog it meant one product with hundreds of variants.
The handler did a fixed amount of work per related row. Cheap at five rows. Ruinous at five hundred. Latency was never a property of the code, it was a function of the data shape the caller happened to have.
Which is why dashboards missed it. They average across users whose data does not look alike, and the shape that breaks you is a minority by definition.
So now I log cardinality next to duration: how many rows, children or positions a response actually assembled. Then I plot latency against size instead of against time. A flat line means fixed cost. A slope means per-row work you will meet again, larger, later.
The slope is the bug. The seconds are only the symptom.
The uncomfortable part is that this failure mode scales with success. Your heaviest, longest-tenured, most valuable accounts hit it first, and they are the ones least likely to file a support ticket about it.
I wrote up more of these slow-response post-mortems, including the ones that hid the longest, here: https://polycratia.com/c/682ba0de
For you it might have been a specific tenant, or a payload size, or a dimension you were not logging at all...