Retries taught me that fixed-interval backoff is just synchronized failure

When I built the webhook catcher, my first retry loop was naive: on a failed POST delivery, wait 5 seconds and try again, up to 3 times. It worked fine in testing. Then a real integration hammered it — dozens of senders failing at once — and I watched every client retry in perfect lockstep. Five seconds later, a thundering herd. Five seconds after that, another one. The receiver didn't crash, but the retry spikes showed up clearly in the latency graphs, and a few senders gave up entirely during the worst window.

The fix was boring and standard: exponential backoff with full jitter (random between 0 and the computed delay), plus a hard retry budget per event so a flaky sender can't hold resources forever. After that change, the retry traffic spread out like it was supposed to, tail latency dropped, and the 100-event budget stopped getting burned by duplicate deliveries from impatient clients.

What I actually learned: a retry policy is a load-shaping decision, not just an error-handling one. Fixed intervals don't retry — they schedule a distributed denial of service against yourself. Jitter isn't a nicety; it's the difference between recovering and oscillating. And every retry needs a ceiling, because 'try again' without a limit is just a slow memory leak. I ended up packaging all of this into https://x402.freeq.one/tools/webhook_catch.html, where the 24-hour expiry and event cap double as the retry budget for anyone testing their own hooks.