When Your Data Breaks on AWS (HA & DR Part 1): Aurora Backtrack, Idempotency, and the Delta Problem

It is 2:47 PM. You run a quick Node.js script to update the price of one product in a supermarket inventory database. The script finishes. Then your stomach drops.

You forgot the WHERE clause.

Every single product in the database now has the same price.

Most of us have either done this or watched a teammate do it. The real question is not whether something will go wrong, but what you do in the next five minutes. Amazon's CTO Werner Vogels put it simply: "Everything fails, all the time."

While preparing a community session on AWS resilience, I kept running into questions that the usual "enable backups and you're fine" advice never answers. This article is my attempt to answer them clearly, in two parts:

• Part 1 (this article): how you undo a bad write, how you stop the same request from being processed twice, and what happens to the data written while you are recovering.
• Part 2: RTO and RPO in depth, DynamoDB Point-in-Time Recovery, whether you should automate restores, and the four AWS disaster recovery strategies.

First, Two Words You Need: HA and DR

These two get mixed up all the time, so let's separate them before going further.

High Availability (HA) keeps a failure from becoming an outage. A server dies, an Availability Zone goes dark, and your users never notice because traffic is already flowing somewhere healthy.

Disaster Recovery (DR) is the plan for when the outage, or the data loss, happens anyway. A whole Region has problems, or someone deletes a production table.

Here is the part that surprised me: HA does not protect you from bad data. If your database has a standby copy in another Availability Zone and you run a broken UPDATE, that broken update is faithfully copied to the standby within milliseconds. Replication copies mistakes too. That is a DR problem, and it is exactly what this article is about.

Part A: Two Ways to Undo a Bad Write in Aurora

Go back to our supermarket disaster. If your database is Amazon Aurora, you have two main options.

Option 1: Point-in-Time Restore (the traditional way)

You tell AWS: "Give me the database exactly as it was at 2:46:59 PM."

AWS does this by provisioning a brand-new cluster and restoring your data into it. When it is ready, you update your backend's connection string (environment variables, secrets, config) to point to the new endpoint. Depending on the size of the database, the whole process can take anywhere from 30 minutes to several hours.

Option 2: Aurora Backtrack (the "undo" button)

Backtrack skips all of that. Instead of building a new database, it rewinds your existing cluster in place to a specific second in the past, up to 72 hours back. It works by reversing changes directly at the storage level.

Because it is the same cluster, the endpoint never changes. No new connection strings, no redeploys. The trade-off is that the database pauses briefly and drops all active connections while it rewinds.

Back to our story: instead of waiting hours for a new cluster, you backtrack to two minutes before the script ran. Within minutes the prices are back to normal, and your app reconnects to the same endpoint as if nothing happened.

A few things worth knowing before you rely on it:

• Backtrack is available for Aurora MySQL-compatible clusters, not Aurora PostgreSQL.
• You have to turn it on by setting a backtrack window (up to 72 hours) when you create the cluster or restore it. You cannot switch it on after the incident.
• It adds a small cost, because Aurora has to keep the change records for that window.

The question that bothered me

Here is where I got stuck. If your application cannot tolerate any downtime, Backtrack looks like the wrong choice, because it pauses the database. But a Point-in-Time Restore takes far longer overall. So which one actually causes less disruption?

The answer is that the two options are protecting different things: availability versus data accuracy.

With a snapshot restore, your app is technically "up" the whole time, but it is up and wrong. Customers are seeing the broken prices until you finally swap the endpoint.

With Backtrack, the app is briefly down, then it is right.

So when is Backtrack the wrong choice? Only when you have a strict rule that the database must never drop active connections. Think of a real-time healthcare data pipeline streaming patient vitals. In those rare cases, teams keep the corrupted database online and fix the data another way. For the vast majority of applications, Backtrack is the faster and smarter option.

Part B: Idempotency, or "Once and Only Once"

The supermarket problem was one bad write. The next problem is subtler: the same good write happening twice.

Why this happens in serverless apps

Picture a customer ordering food on a restaurant app and tapping Pay.

1. The frontend sends the checkout request.
2. Your payment Lambda charges the card. The charge succeeds.
3. A brief network drop means the success response never reaches the phone.
4. The frontend times out and automatically retries the exact same request.

Without protection, your Lambda charges the customer a second time. Nothing crashed, nothing is "broken", and yet you now have an angry customer and a refund to process.

Idempotency is the fix. An operation is idempotent when receiving the same request many times has the same effect as receiving it once. The first request does the work. Every duplicate simply gets the original response back.

How Powertools for AWS Lambda handles it

Powertools for AWS Lambda has an idempotency utility that uses a fast key-value store, usually DynamoDB, as a receipt ledger.

When a request arrives, Powertools pulls a unique value out of it, called the idempotency key, and checks the ledger:

• The key is new: Powertools records it in DynamoDB as "in progress", lets your function run, then saves the final response against that key.
• The key already exists: Powertools recognizes a retry, skips your business logic completely, and returns the saved response.

In the restaurant example, the Order ID is the key. On the retry, Powertools sees the order is already completed and returns 200 OK without ever calling the payment provider again. The customer is charged exactly once.

Here is what that looks like in TypeScript:

import { makeIdempotent, IdempotencyConfig } from '@aws-lambda-powertools/idempotency'; import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb'; // The DynamoDB table acting as our receipt ledger const persistenceStore = new DynamoDBPersistenceLayer({ tableName: 'IdempotencyTable', }); // Use the orderId from the request body as the idempotency key const config = new IdempotencyConfig({ eventKeyJmesPath: 'powertools_json(body).orderId', }); const processPayment = async (event: any) => { const { orderId, amount } = JSON.parse(event.body); // Your real payment logic goes here. // On a retry with the same orderId, this code never runs. return { statusCode: 200, body: JSON.stringify({ message: 'Payment successful', orderId }), }; }; export const handler = makeIdempotent(processPayment, { persistenceStore, config, }); 

The idempotency table needs a partition key named id (String). It is also a good idea to enable TTL on an expiration attribute so old receipts clean themselves up.

Questions people usually ask next

Is the idempotency key the same thing as the Order ID?

Often, yes. If a natural unique ID already exists, like an orderId, it makes a perfect idempotency key. But some requests do not have one yet. Think of a "Create Account" form: the database has not assigned an ID because the account does not exist. In that case the key is created in one of two ways:

1. Client-side generation: the frontend generates a random UUID and sends it in a header, for example Idempotency-Key: 123e4567-e89b-.... Every retry reuses the same UUID.
2. Payload hashing: Powertools runs the request body through a hash function, and the resulting hash becomes the key. Identical payloads produce identical keys.

In short, the key is simply whatever string your backend uses as the lock to decide "have I already done this?"

What makes DynamoDB a "fast key-value store", and why use it here?

A key-value store is a NoSQL database built for one thing: very fast, very simple lookups. There are no joins, foreign keys or complex schemas. You store a key (the idempotency token) and a value (the status and saved response), much like a dictionary. DynamoDB and Redis are the common examples. Reading a single key takes single-digit milliseconds, so checking the ledger on every request barely adds anything to your API response time.

What does "state-changing action" mean?

It is any operation that changes the data, or "state", of your system:

• State-changing: creating, updating or deleting data (POST, PUT, PATCH, DELETE). Writing a record, reducing a stock count, processing a payment.
• Not state-changing: reading data (GET). Listing users does not change the database.

Idempotency exists to protect state-changing actions. If a GET runs twice, the user just sees the same data twice. If a payment POST runs twice, someone gets charged double.

Part C: The Delta Problem (the Hardest Part of Database Recovery)

Now back to the Point-in-Time Restore from Part A, because it hides a problem that many tutorials skip entirely.

What actually happens during a restore

Say the corruption happens at 14:02. You start a restore to 14:01:59. AWS spends the next hour or so building the new cluster. Meanwhile, your original database is still online and still accepting writes. New orders, new patient records and new payments keep arriving, and they are perfectly valid.

At 15:10 the new cluster is ready and you swap the endpoint.

So where are the 68 minutes of valid writes that landed on the old database during the restore?

They are still in the old database. AWS does not move them for you.

The new cluster only contains data up to 14:01:59. Everything written after that, the good along with the bad, stays behind. Database engineers call this gap the Delta Problem or the Cutover Window.

Here are the three ways teams deal with it, from the most common to the most manual.

Strategy 1: Read-Only Freeze and Queue (safest and most common)

The moment you discover the corruption, switch the old database to read-only.

• Reads (GET) keep working, so users can still view their data.
• Writes (POST, PUT, DELETE) are rejected by the database.

To avoid losing those writes and showing users errors, your API catches the failed writes and parks the payloads in an Amazon SQS queue. Once the new cluster is live and the endpoint is swapped, a Lambda worker drains the queue and writes every parked request into the new database.

This is also where Part B pays off. If any of those queued messages get delivered twice, idempotent handlers make sure nothing is written twice.

Strategy 2: Change Data Capture with AWS DMS (the advanced AWS way)

If the application absolutely must stay fully read-write and queuing is not an option, teams use AWS Database Migration Service (DMS) as a live bridge between the old database and the new one.

DMS supports Change Data Capture (CDC): it reads the database transaction log (the binlog in MySQL, or the WAL in PostgreSQL) and replays changes on the target. You start CDC from a log position just after the bad transaction, so only the valid inserts and updates made during the restore are streamed into the new cluster. Once the new cluster has caught up to the present, you swap the endpoint.

It is powerful, but it needs careful setup and validation. Writes that touched the same rows as the corrupting statement may still need a human to review them.

Strategy 3: Custom Reconciliation Script (the manual way)

If the disaster hits and you have neither a queue nor DMS in place, you rescue the data by hand.

1. Swap the endpoint so the app uses the new cluster.
2. Keep the old database online, but isolated from the app.
3. Write a script (for example in Node.js with Prisma) that connects to both databases.
4. Query the old database for rows where createdAt or updatedAt falls between the corruption time and the cutover time.
5. Copy those missing rows into the new database.

It works, but stitching data together by hand while a live system is running is stressful and easy to get wrong.

Comparing the three

And this is the real reason Aurora Backtrack is so attractive. Because it rewinds the same database in place, there is no second database and no cutover window to reconcile. The brief pause it causes is often a small price compared to stitching data together under pressure.

Key Takeaways

1. High Availability does not protect you from bad data. Replication copies mistakes too.
2. Point-in-Time Restore keeps the app online but serving wrong data until you swap endpoints. Backtrack causes a short pause but fixes data in place, on the same endpoint.
3. Turn Backtrack on before you need it. It only works on Aurora MySQL-compatible clusters with a backtrack window set.
4. Retries are normal in serverless systems. Make every state-changing action idempotent, especially payments.
5. A restored database does not include writes made during the restore. Plan for the Delta Problem with a read-only freeze and SQS, DMS CDC, or a reconciliation script.

Coming Up in Part 2

In the next article I will cover:

• RTO and RPO: what they really mean, who decides them, and what happens when you miss them
• DynamoDB Point-in-Time Recovery: the 35-day window, per-second restores, and why it never overwrites your table
• Should you automate database restores? Why it sounds smart and why it is usually a bad idea
• The four AWS DR strategies: backup and restore, pilot light, warm standby, and active/active

If you have ever had to recover from a bad write in production, I would love to hear how you handled it. Drop it in the comments.