Our .NET App Crashed on 11.11. Here's the Queue Pattern That Fixed it

1. The Monday brief

I joined Lion City Logistics, a delivery startup in Singapore. Three weeks earlier, their tracking app (ParcelTrack) went down for hours during the 11.11 sale.

On my first Monday, the CTO sent me this message:

"Move ParcelTrack to AWS Singapore. It must handle traffic spikes. Keep secrets out of the code. Tell us when something breaks, before customers notice. Deploy automatically when we merge to main on Bitbucket. We need a UAT environment and a production environment."

First, I wanted to know what actually broke. If I move a broken design to the cloud, I just get a broken design that costs more money.

2. What went wrong

I read the old code and the incident notes. I found five problems.

Problems 1 and 2 caused the crash. The other three made it worse and made recovery hard.

3. The one rule I decided to follow

I wrote one sentence on a sticky note and put it on my monitor:

The API only does fast things. Anything slow, heavy, or likely to fail goes on a queue. A separate worker does it.

4. Why I ran the app on my laptop first

I could have started clicking in AWS right away. But I have learned this the hard way: if you debug code problems and cloud problems at the same time, you cannot tell which one is failing.

So first I made the new design work on my laptop. Then I moved it to AWS, knowing the code was fine.

To do this, I run six containers. Some were new to me, so here is what each one is.

The trick that makes this work

I want the same code on my laptop and in AWS. So I use one setting to choose where my code talks to. I show it in Stage 3.

5. Get the code

The folder looks like this:

parceltrack-aws-tutorial/ ├── src/ │ ├── ParcelTrack.Core/ shared model, database, email, AWS setup │ ├── ParcelTrack.Api/ the web API (fast things only) │ ├── ParcelTrack.Worker/ background jobs (slow things) │ └── ParcelTrack.Web/ the web page (Blazor) ├── scripts/ LocalStack setup + a quick test ├── docker-compose.yml starts all six containers └── README.md 

To run everything:

docker compose up --build 

The first build takes a few minutes. Then open:

At the end of each stage, I can run something and see it work. If a stage breaks, I know which new piece caused it.

6. Building it in five stages

Stage 1: The database and the API

Goal: create a parcel and list parcels. Nothing else yet.

Files: Core/Parcel.cs, Core/ParcelDb.cs, Api/Program.cs, Api/Dockerfile

I put the data model in a separate Core project. Both the API and the worker need to read the same tables. If I define a parcel in one place, the two cannot disagree.

This C# class becomes a database table:

public class Parcel { public Guid Id { get; set; } public string TrackingNumber { get; set; } = ""; public string Recipient { get; set; } = ""; public string CustomerEmail { get; set; } = ""; public string Status { get; set; } = "Created"; // Created -> InTransit -> Delivered public DateTime CreatedAt { get; set; } public DateTime UpdatedAt { get; set; } } 

This is how Entity Framework talks to the database. DbSet<Parcel> means "a table of parcels":

public class ParcelDb(DbContextOptions<ParcelDb> options) : DbContext(options) { public DbSet<Parcel> Parcels => Set<Parcel>(); public DbSet<Invoice> Invoices => Set<Invoice>(); } 

In the API's Program.cs, the first lines connect to the database:

builder.Services.AddDbContext<ParcelDb>(o => o.UseNpgsql(builder.Configuration.GetConnectionString("Default"))); 

Look at where the connection string comes from. It is not in the code. It comes from configuration. This is my first step to fix problem 4 (the password in the repo).

Then there are the endpoints. The important ones:

app.MapHealthChecks("/health"); // "Am I alive?" The load balancer will ask this later. app.MapGet("/api/parcels", ...); // list parcels app.MapPost("/api/parcels", ...); // create a parcel 

The Dockerfile uses two stages. The first stage has the full .NET SDK and builds the app. The second stage copies only the result onto a small image. The final image is much smaller and has no compiler in it.

I copy the .csproj files and run restore before I copy the source code. Docker saves that step in its cache, so later builds take seconds.

ASP.NET Core and environment variables: in docker-compose.yml you see ConnectionStrings__Default. The double underscore __ means a nested setting. It becomes ConnectionStrings:Default. ECS and EKS will use the same method later.

Check it works:

curl -X POST localhost:8080/api/parcels \ -H "Content-Type: application/json" \ -d '{"recipient":"Tan Wei Ming","email":"tan@example.com"}' curl localhost:8080/api/parcels 

You should see your parcel with a tracking number like LCL482910.

Stage 2: The web page (what the user sees)

Goal: a page that talks to the API.

Files: Web/Program.cs, Web/Pages/Home.razor, Web/wwwroot/appsettings.json

I used Blazor WebAssembly. It is C#, which my team already knows. It also builds into plain static files. Static files need no server. This will be useful when I use S3 and CloudFront later.

Three things matter on this page.

1. The API address is a setting, not hard-coded. It lives in wwwroot/appsettings.json:

{ "ApiUrl": "http://localhost:8080" } 

Later, I can point the same build at UAT or production by changing this one file.

2. The page refreshes by itself every 3 seconds. So you can watch the status change:

timer = new Timer(async _ => { await Load(); await InvokeAsync(StateHasChanged); }, null, 3000, 3000); 

3. The button that creates a parcel:

var res = await Http.PostAsJsonAsync("api/parcels", new { Recipient = recipient, Email = email }); 

My first problem here: CORS. My browser refused to let the page on localhost:8081 call the API on localhost:8080. This is a browser safety rule. The API must specify which websites are allowed. I fixed it in the API:

builder.Services.AddCors(o => o.AddDefaultPolicy(p => p.WithOrigins(origins).AllowAnyHeader().AllowAnyMethod())); // ... app.UseCors(); 

origins comes from a setting (AllowedOrigins). UAT and production will have different website addresses, so I did not want to hard-code it.

Check it works: open http://localhost:8081, type a name and email, and click Create parcel. A new row appears.

Stage 4: The worker

Goal: read the tickets, make the PDF, store it in S3, and email the link.

Files: Worker/InvoiceWorker.cs, Worker/InvoicePdf.cs, Core/Email.cs, Worker/ParcelProgressWorker.cs

A .NET Worker Service is a console app that runs background tasks. The template gives you a Worker.cs that only prints the time. I replaced it with InvoiceWorker. It has two parts.

Part 1: the loop that takes tickets

var res = await sqs.ReceiveMessageAsync(new ReceiveMessageRequest { QueueUrl = queueUrl, MaxNumberOfMessages = 5, WaitTimeSeconds = 20 // long polling }, ct); foreach (var msg in res.Messages) { try { var job = JsonSerializer.Deserialize<InvoiceRequested>(msg.Body)!; await Process(job.InvoiceId, ct); await sqs.DeleteMessageAsync(queueUrl, msg.ReceiptHandle, ct); // success: remove the ticket } catch (Exception ex) { log.LogError(ex, "Invoice job failed"); // do NOT delete: it will come back and retry } } 

Two things to notice:

• Long polling (WaitTimeSeconds = 20): SQS waits up to 20 seconds for a message before it answers "nothing". This means fewer empty calls and a smaller bill.
• I delete the message only after success. This one rule is the whole retry system. No delete means the message comes back. Part 2: the real work

async Task Process(Guid invoiceId, CancellationToken ct) { var inv = await db.Invoices.FindAsync([invoiceId], ct); if (inv.Status == "Sent") return; // see "idempotency" below // 1. Make the PDF and store it in S3 var pdf = InvoicePdf.Render(inv); // ... upload to S3, set Status = "Generated" // 2. Make a link that works for 24 hours, and email it var link = s3.GetPreSignedURL(/* bucket, key, expires in 24h */); await email.SendAsync(inv.CustomerEmail, "Your invoice", $"Download: {link}", ct); // 3. Mark as Sent } 

Two new words I learned here:

• Idempotency. SQS says "at least once". Sometimes the same message arrives twice. Without the Status == "Sent" check, the customer would get two emails. Every worker should be safe when the same job runs twice.
• Pre-signed URL. A temporary link that lets someone download one private file for a short time. The bucket stays private. The customer still gets a working link. Swappable email. My worker calls email.SendAsync(...). It never says which email system. There are two classes behind one interface, and a setting picks one:

s.AddSingleton<IEmailSender>(sp => cfg["Email:Provider"] == "ses" ? ActivatorUtilities.CreateInstance<SesEmailSender>(sp) // AWS (Tutorial 3) : ActivatorUtilities.CreateInstance<SmtpEmailSender>(sp)); // Mailpit (my laptop) 

This is the same idea as LocalStack: the worker code does not change between my laptop and AWS.

The simulator. There are no real drivers in my demo. So I added a second small worker, ParcelProgressWorker. It moves parcels from Created to InTransit to Delivered every few seconds. This makes the page look alive.

It is not part of the real design, so I put it behind a switch:

if (builder.Configuration.GetValue<bool>("Simulator:Enabled")) builder.Services.AddHostedService<ParcelProgressWorker>(); 

I turn it on locally and in UAT. I leave it off in production, where real driver updates will come in.

One more fix I made. At first, the worker crashed if LocalStack was not ready yet. Now the worker catches the error, waits 5 seconds, and tries again.

Stage 5: Put it all together in Docker Compose

Six services in one file is a lot to read. So I wrote the settings that the API and worker share just once:

x-app-env: &app-env # a reusable block  ConnectionStrings__Default: "Host=db;Database=parceltrack;Username=postgres;Password=localdev" Aws__ServiceUrl: http://localstack:4566 # "talk to the fake AWS"  AWS_ACCESS_KEY_ID: test # LocalStack accepts any key  AWS_SECRET_ACCESS_KEY: test AWS_REGION: us-east-1 InvoiceQueueUrl: http://localstack:4566/000000000000/parceltrack-invoices InvoiceBucket: parceltrack-invoices-local Email__Provider: smtp # "use Mailpit, not SES"  Smtp__Host: mailpit Smtp__Port: "1025" 

Each service then reuses the block:

 api: environment: *app-env worker: environment: <<: *app-env # everything above...  Simulator__Enabled: "true" # ...plus one more setting 

One thing that confused me: inside Docker Compose, containers find each other by service name. So the API connects to Host=db and http://localstack:4566. It never uses localhost. Inside a container, localhost means the container itself.

7. The 11.11 test

Now I tested the exact thing that broke before.

First the normal flow:

1. Create a parcel on the page.
2. Click Email invoice.
3. Watch the badge go Pending then Generated then Sent.
4. Open http://localhost:8025. The email is there. Now the disaster:

docker compose stop mailpit # the email server "dies" 

I created and invoiced 20 parcels. The page stayed fast, because the API only writes tickets. The worker printed errors. The tickets waited in the queue.

Then:

docker compose start mailpit # the email server comes back 

In about a minute, all 20 emails arrived.

In the old system, this exact situation took the whole site down. This is when I felt ready to show my boss.

8. From the brief to an AWS shopping list

Now that the code works, I can choose AWS services. For each need, I asked: what do I need, and which service gives it to me?

This is the target picture:

Why two ways to run containers? The CTO asked for EKS. But ECS Fargate is simpler and cheaper. It has no cluster to manage. EKS is more portable, and many employers ask for it. I will build both and compare them in a later post.

9. UAT and production

My rule: same Terraform code, different settings. If UAT is built differently from production, a passing UAT test means little.

In a real company, I would put UAT and prod in two separate AWS accounts. This is safer. To keep this series cheap, I will use one account with two clearly separate environments. They will have separate networks, separate state files, and uat- and prod- name prefixes. I will point out where you would split accounts.

The Terraform folder will look like this (I build it in the next post):

infra/ ├── bootstrap/ the bucket that stores Terraform state ├── modules/ network, ecr, rds, sqs, ecs, eks ... written once └── envs/ ├── uat/ small and cheap └── prod/ safe and strong 

Next: Tutorial 2

I will set up my AWS account safely (SSO and a budget alarm). I will learn Terraform properly. Then I will build the shared base for UAT and production: remote state, network (VPC), ECR, and S3.