The Key Expires in 24 Hours. The Retry Arrives on Monday.

itsvicmoreno left a comment on my last piece about the crash between the key insert and the charge, and it named the hole my run never tested. My rig crashed the receiver between writing the key and charging the card, in both orders, and measured what each shape lost. But every retry in that run arrived seconds later, while the provider still held the key. itsvicmoreno asked about the retry that arrives after the key is gone.
The provider's own docs say an idempotency key can be pruned once it is at least 24 hours old. A retry that waits past that, a dead-letter replay after a weekend outage or a row reclaimed the next day, is treated as a brand-new request. The shape as I ran it never tested a retry outside the window. So I built the weekend: same event, same $29 payment for order ORD-1042, a crash after the first charge lands, and then the retry arriving 25 hours later, after the key has expired. Two candidate fixes, and one of them survives. The other one refuses the retry and leaves the order stuck.
My claim before the numbers: the retry-age cap would be the fix, because refusing a stale retry is simpler than teaching the retry to look anything up. I was wrong about which one holds.
Problem 1: the expired key charges twice, exactly as the docs say
The control arm is the shape most systems actually have. First attempt: the charge lands at the provider, then the receiver dies before marking it done. Twenty-five hours later the dead-letter queue replays it. The provider's key is 90,000 seconds old. The lifetime is 86,400. The provider no longer holds it, so the retry looks like a new request, and the provider charges again.
Two provider charges. $58.00 for a $29.00 order. One local charge on the books, one charge the customer sees twice and nobody on our side can explain from the local record alone, because the local record says the order was charged once. The retry returned 200, status processed, and nothing in that response says anything went wrong. The double charge is invisible until the customer finds it, which is the same asymmetry as the last piece: a false "already charged" costs one charge nobody sees, and a false "not charged" costs a double charge somebody sees. This one is the second kind, and it still took a customer to notice, because our side of the record looked clean.
I ran the whole thing twice, and both runs gave the same numbers in every arm. The double is not a flake. It is what the shape does.
Problem 2: the order-id lookup makes the provider's clock irrelevant
The second arm changes one thing. Before retrying, the receiver searches the provider by our own order id, ORD-1042, instead of asking by the key. The key is still expired. The lookup does not care. The provider finds the first charge under that order id, answers "already charged", and the retry stops there.
One provider charge. $29.00. The retry returns 200, status duplicate, detail "provider already charged (order-id lookup)", and nothing is charged a second time. The order lands in "done" on the local record, which is the part the other fix never manages.
itsvicmoreno said it first in his follow-up: search the provider by your own order id before retrying, and the provider's clock stops mattering. He was right, and the run proves it in the shape that matters most: the retry that arrives after the key is dead is exactly the retry the lookup saves. The key is a convenience for the first 24 hours. The order id is the record that outlives it. If your retry path only knows the key, it has a 24-hour memory and a weekend-long blind spot.
Problem 3: the age cap refuses in time, and abandons the order
The third arm is the fix I believed in before the run. The retry path refuses any retry older than a 23-hour cap, set below the provider's 24-hour lifetime. The replay arrives at 25 hours. The receiver refuses it with a 422: retry older than cap, needs manual reconciliation.
One provider charge. $29.00. No double. The money is safe. But look at the local record: the order is still sitting in "processing". Zero local charges. The customer was charged once, the order was never marked done, and the only thing standing between that row and a human is a dead-letter entry that says "needs manual reconciliation". The cap protected the money and abandoned the job. Somebody has to go and finish it by hand, and if nobody does, the customer paid for an order the system still thinks is unfinished.
That is the honest shape of the cap. It is a backstop, not a fix. It stops the bleeding and leaves the wound open. On a quiet week, one manual reconciliation is nothing. On a weekend outage with a queue full of replays, it is a Monday morning of manual work, and every row in that queue is a customer who already paid.
Problem 4: the two fixes are not symmetric, and the order matters
So which one holds? The lookup, and it is not close. The lookup finishes the order: one charge, done on the record, no human needed. The cap refuses the retry: one charge, processing on the record, human needed. Both keep the money at $29. Only one finishes the work.
The right shape is both, in this order: lookup first, cap as the backstop. The lookup handles the replay that arrives after the key is gone, which is the common case after an outage. The cap handles the replay the lookup cannot settle — the provider that cannot be searched by order id, the lookup that fails, the answer that comes back "unknown". The cap is the fence at the edge. The lookup is the road.
The limit I have to state plainly: the provider here is a second table in the same process, modelling exactly one property — an idempotency-key lookup with a 24-hour lifetime. It has no network failures, no provider-side timeouts, no partial provider state. The 25-hour gap was simulated by ageing the stored key in place, not by waiting a day. What the run proves is the shape of the three arms against a key that is gone, not the behaviour of any real provider's prune. And the lookup only works if the provider lets you search by your own order id. If yours does not, the cap is all you have, and you should know what it leaves behind.
What do you do with retries older than the key lifetime: cap them, look them up, or let them through? I would like to hear which one your system does, and whether you found out from the design or from the second charge.
P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam
原文:https://www.a8q9.cn/en/article/the-key-expires-in-24-hours-the-retry-arrives-on-monday-0ed828.html
