The callback arrived and the signature check rejected it. I replayed that exact payload on my laptop with the same secret, and it verified fine.
Same code. Same key. Same bytes. Different answer.
The provider signs the request together with a timestamp, and verification allows a tolerance window around that timestamp. My laptop's clock was synced. The box running that worker was not. Time had drifted far enough that legitimate callbacks looked like replays of something too old.
Nothing in the code could point at this, because the code was correct.
What it cost: a stretch of dropped payment notifications on one worker while the others kept going normally. Money that had settled already sat unacknowledged on my side until reconciliation caught up. Intermittent and partial, and invisible in application logs: the worst shape a payment bug can take.
Now I treat the clock as a dependency instead of ambient truth. Anywhere a signed timestamp is verified, I record the delta between the provider's timestamp and local receipt time, and I alert on that delta long before it reaches the tolerance edge. Skew becomes a graph instead of a mystery.
If you verify signed timestamps, I'd go look at how wide your tolerance window is, and whether anything tells you when you are drifting toward its edge...