3 seconds to 200ms usually says nothing about speed. It says how often requests now take the fast path.
Most endpoints optimized this way end up bimodal: a cheap path that reads something already computed, and the original expensive path, unchanged, still sitting behind a cache miss, a cold key, an invalidation. The median moved because the mix moved. Nothing got faster.
That distinction matters, because the two failure modes are not equivalent.
A flat 3 seconds is slow but survivable. The load it puts on the database or the upstream provider is constant, and you have already capacity-planned for it.
A 200ms median with a surviving 3-second path concentrates that same work into the moments when the fast path is empty: right after a deploy, after a bulk catalog import rewrites half the keys, after a provider outage clears and every stale entry expires in the same minute. You traded steady slowness for a correlated stall, and the stall shows up exactly when the rest of the system is already degraded.
So before I call it a win, I measure three things separately. The cost of the slow path on its own. The share of traffic that still takes it. And whether the service stays up if that share goes to 100 percent for a few minutes.
If the answer to the third one is no, you don't have a 200ms endpoint. You have a 3-second endpoint with good weather.
The fix is unglamorous: single-flight on the expensive path, so a thousand simultaneous misses cost one computation instead of a thousand, and staggered expiry so keys don't die in formation.
I write up more of these trade-offs here: https://polycratia.com/c/e7cc6a9c
If you have shipped this kind of rewrite, I'd look at what your p99 does in the first minute after a deploy...