Resilient Actions

One night, Cron was generating jobs as expected. The queue was accepting them as expected. The Worker was running as expected.

Then a single Action got stuck.

As a result:

• Cron continued generating new jobs;
• the queue continued growing;
• other Actions stopped executing;
• the issue was resolved only after a restart.

This is a classic example of how system reliability is determined not by how it behaves in a perfect world, but by how it handles failures.

An Action Must Be Designed for Failure

Every Action operates in a hostile environment:

• the network may stop responding;
• the database may become slow;
• an external API may hang;
• the server may restart;
• the action may be cancelled by the user.

For this reason, an Action should be designed as if failure is inevitable.

Always Use CancellationToken

Bad:

await http.GetStringAsync(url); 

Good:

await http.GetStringAsync(url, ct); 

Bad:

await Task.Delay(TimeSpan.FromMinutes(5)); 

Good:

await Task.Delay(TimeSpan.FromMinutes(5), ct); 

If an operation does not support cancellation, the Worker cannot stop the Action gracefully.

Always Define Timeouts

Even when a CancellationToken is used.

Bad:

var http = new HttpClient(); 

Good:

var http = new HttpClient { Timeout = TimeSpan.FromSeconds(30) }; 

External resources are not required to respond quickly.

Sometimes they do not respond at all.

Log Execution Stages

Often the most useful log messages are the simplest ones:

ReportProgress(0, "Loading data"); 

ReportProgress(0, "Downloaded data"); 

ReportProgress(0, "Saving data"); 

ReportProgress(0, "Completed"); 

If an Action gets stuck, the last message immediately identifies where execution stopped.

For example:

Loading new 

It becomes obvious that the problem occurred before the database save phase.

Make Actions Idempotent

After a failure, an Action may execute again.

If re-execution creates duplicate data, the system is not resilient.

Bad:

INSERT INTO Tokens (...) 

Better:

INSERT ... WHERE NOT EXISTS (...) 

or

MERGE 

or

UNIQUE INDEX 

A repeated execution should be safe.

Do Not Store State in Memory

Bad:

private static int Processed; 

The value will be lost after a restart.

Instead, store state in:

• a database;
• a queue;
• a persistent state store.

An Action should survive process restarts.

Break Large Operations into Smaller Actions

Bad:

DownloadAllHistory(); ProcessAllHistory(); SaveAllHistory(); 

Good:

download process aggregate publish 

Smaller Actions are easier to:

• cancel;
• retry;
• troubleshoot.

Verify Behavior After a Timeout

It is not enough to verify that an Action was cancelled.

You must prove that the queue continues to operate.

A good integration test is:

1. The first job becomes stuck.
2. A timeout occurs.
3. The second job executes successfully.
4. The queue continues processing.

This scenario closely matches real production behavior.

A Practical Rule

Whenever an Action contains an:

await 

ask yourself:

What happens if this operation hangs forever?

If the answer is:

Nothing good.

Then the Action is not production-ready yet.

Summary

A simple, resilient Action has five characteristics:

• supports CancellationToken;
• defines reasonable timeouts;
• logs key execution stages;
• is idempotent;
• safely handles re-execution.

In an ideal world, failures never happen.

In production, they happen every day.

A good Action is not one that never fails.

A good Action is one after which the system continues to operate. 🚀🐈‍⬛