Handling Redis Job Queue Poison Pills: Implementing Dead Letter Queues (DLQ) & Exponential Backoff in BullMQ

Most job-queue incidents don't start with an outage. They start with one bad job.

A webhook payload with a field your code never expected. A user record that triggers an edge case in your PDF renderer. A third-party API that permanently rejects one specific request. The job fails, gets retried, fails again, and keeps failing, burning worker capacity, spamming your logs, firing the same alert 400 times, and making it hard to see the real problems.

Messages like this are called poison pills: input that can never be processed successfully, no matter how many times you try. And there's a second, nastier variety, the job that crashes the worker process itself (out-of-memory, infinite loop, native-module segfault), which gets retried onto the next worker and crashes that one too.

In this post we'll build a defense in BullMQ with four layers:

1. Classify errors so permanent failures don't waste retries
2. Retry transient errors with exponential backoff and jitter
3. Park exhausted jobs in a dead letter queue (DLQ) with full context
4. Alert and replay deliberately once the cause is fixed

What BullMQ gives you (and what it doesn't)

BullMQ has the building blocks, but not a ready-made DLQ:

That failed set is already a de-facto dead letter store. Many teams stop there, and that's a legitimate choice. We'll build an explicit DLQ because it gives you a separate place with separate retention, separate alerting, and clean replay tooling, and because it protects failed jobs from being swept away by your normal removeOnFail cleanup. We'll compare both options near the end.

Step 1: Configure attempts and backoff

import { Queue } from 'bullmq'; export const emailQueue = new Queue('emails', { connection, defaultJobOptions: { attempts: 5, backoff: { type: 'exponential', delay: 1_000 }, // 1s, 2s, 4s, 8s between attempts removeOnComplete: { age: 3600, count: 1000 }, removeOnFail: { age: 14 * 24 * 3600 }, // keep terminal failures for 14 days }, }); 

With type: 'exponential' the wait before each retry doubles, starting from delay. That's the right shape for transient failures: the first retry is quick (most blips clear in seconds), and later retries back off so you don't hammer a struggling dependency.

Why backoff alone isn't enough: add jitter

If 5,000 jobs fail at the same moment because a downstream service went down, plain exponential backoff retries all 5,000 in lockstep at 1s, then 2s, then 4s, a self-inflicted thundering herd each time. Jitter randomizes each job's delay so retries spread out.

Recent BullMQ versions support a jitter option on the backoff setting, so check your installed version's docs. If yours doesn't, or you want a cap on the maximum delay, use a custom backoff strategy on the worker:

const BASE_MS = 1_000; const MAX_MS = 5 * 60_000; // never wait longer than 5 minutes const worker = new Worker('emails', processor, { connection, concurrency: 10, settings: { // Full jitter: random delay between 0 and the capped exponential value backoffStrategy: (attemptsMade: number) => { const exp = Math.min(MAX_MS, BASE_MS * 2 ** (attemptsMade - 1)); return Math.round(Math.random() * exp); }, }, }); // jobs opt in with a custom backoff type await emailQueue.add('welcome', data, { attempts: 6, backoff: { type: 'custom' } }); 

(The exact settings.backoffStrategy signature has evolved between BullMQ versions, so check the docs for the version you run.) Full jitter, a random value between zero and the exponential ceiling, is a good default because it spreads load most evenly. The trade-off is that an individual retry can be very quick.

Two more details worth setting deliberately:

• A cap on delay. Unbounded exponential growth makes late attempts absurdly long: ten doublings of one second is 17 minutes.
• A sensible attempt count. Five to eight attempts with backoff covers outages lasting minutes. If a dependency is down for hours, retries aren't the right tool. You want the circuit-breaker and DLQ-replay approach below.

Step 2: Classify errors. Not every failure deserves a retry

Retrying an error that can never succeed is pure waste. Split failures into two classes:

BullMQ provides UnrecoverableError for the second class. Throwing it skips remaining attempts and moves the job straight to failed:

import { Worker, UnrecoverableError } from 'bullmq'; import { z } from 'zod'; const WelcomeEmail = z.object({ userId: z.string().uuid(), template: z.enum(['welcome', 'reminder']), }); const processor = async (job: Job) => { // 1. Validate input. A bad payload will never become a good payload. const parsed = WelcomeEmail.safeParse(job.data); if (!parsed.success) { throw new UnrecoverableError(`Invalid payload: ${parsed.error.message}`); } const user = await db.users.find(parsed.data.userId); if (!user) throw new UnrecoverableError(`User ${parsed.data.userId} not found`); // 2. Call the outside world and classify its failures try { await mailer.send(user.email, parsed.data.template, { signal: AbortSignal.timeout(10_000) }); } catch (err) { throw classify(err); } }; function classify(err: any): Error { const status = err?.status ?? err?.response?.status; if (status === 429) return err; // transient: retry with backoff if (status >= 400 && status < 500) { return new UnrecoverableError(`Provider rejected request (${status}): ${err.message}`); } return err; // 5xx, network errors, timeouts: retry } 

When in doubt, default to retrying. Misclassifying a transient failure as permanent loses work, while misclassifying a permanent one as transient just wastes a few attempts.

Respect rate limits properly

For 429 responses with a Retry-After, BullMQ has a purpose-built mechanism that pauses the whole worker rather than burning individual job attempts:

if (status === 429) { const waitMs = Number(err.headers?.['retry-after'] ?? 1) * 1000; await worker.rateLimit(waitMs); throw Worker.RateLimitError(); // job goes back to the queue without counting a failure } 

(Check the BullMQ docs for the exact rate-limit API in your version.)

Step 3: Add a per-job timeout

BullMQ doesn't time out your processor. A job stuck on a hung socket holds a concurrency slot indefinitely, and as long as the worker's event loop is alive it keeps renewing the job's lock, so the job never even becomes "stalled." Ten hung jobs on a concurrency-10 worker means zero throughput with no errors.

Add the timeout yourself, and pass the signal down to everything that supports it:

async function withTimeout<T>(fn: (signal: AbortSignal) => Promise<T>, ms: number): Promise<T> { const signal = AbortSignal.timeout(ms); return Promise.race([ fn(signal), new Promise<never>((_, reject) => signal.addEventListener('abort', () => reject(new Error(`Job timed out after ${ms}ms`)), { once: true }), ), ]); } const processor = (job: Job) => withTimeout((signal) => handle(job, signal), 30_000); 

A timeout is a transient failure (the next attempt may succeed), so it uses normal retry and backoff. Note that Promise.race only stops waiting. Work that ignores the abort signal keeps running, which is why passing the signal into fetch, database clients, and so on matters.

Step 4: The dead letter queue

When a job fails terminally (attempts exhausted, UnrecoverableError, or too many stalls), copy it into a dedicated queue with everything needed to diagnose and replay it.

import { Queue, Worker, type Job } from 'bullmq'; const dlq = new Queue('emails-dlq', { connection, defaultJobOptions: { removeOnComplete: false, removeOnFail: false, // the DLQ is for keeping things; expire deliberately (see below) }, }); worker.on('failed', async (job: Job | undefined, err: Error) => { if (!job) return; // 'failed' fires on EVERY failed attempt. Only act when the job is terminally failed. // If a retry is scheduled, the job is 'delayed' (or 'waiting'), not 'failed'. if ((await job.getState()) !== 'failed') return; await dlq.add( 'dead', { originalQueue: job.queueName, originalJobId: job.id, name: job.name, data: job.data, failedReason: job.failedReason ?? err.message, stacktrace: job.stacktrace, attemptsMade: job.attemptsMade, firstEnqueuedAt: job.timestamp, deadAt: Date.now(), }, // Custom job id => adding the same dead job twice is a no-op (idempotent) { jobId: `dead-${job.queueName}-${job.id}` }, ); log.error({ jobId: job.id, queue: job.queueName, reason: err.message }, 'job moved to DLQ'); }); 

Design points:

• Check the job's state, not just the attempt count. job.getState() === 'failed' is true for exhausted retries, UnrecoverableError, and jobs that stalled too many times, so one check covers all terminal paths. The failed event also fires for attempts that will be retried, and you don't want those in the DLQ.
• A deterministic jobId makes insertion idempotent. If the handler runs twice, BullMQ ignores the duplicate. (Custom job ids can't contain : in BullMQ, so use a different separator, as above.)
• Keep the full context: original payload, error, stack trace, attempt count, and timestamps. A DLQ entry you can't diagnose is just a bigger problem.
• Don't attach a normal worker to the DLQ. Its jobs sit in waiting as an inspectable inventory. Only add a consumer if it's an alerting or replay tool.

The gap: the handler can be lost

The failed event handler runs in the worker process. If the process dies in the instant between the job failing and the DLQ write, that job sits in the failed set and never reaches the DLQ. Two ways to cover it:

1. Run a periodic reconciler that scans the failed set (queue.getFailed(start, end)) and adds any job missing from the DLQ. The deterministic jobId makes this safe to run repeatedly.
2. Treat the failed set as the source of truth (see the comparison section) and use the explicit DLQ only as a convenience view.

export async function reconcileDlq(queue: Queue, dlq: Queue) { const PAGE = 100; for (let start = 0; ; start += PAGE) { const failed = await queue.getFailed(start, start + PAGE - 1); if (failed.length === 0) break; for (const job of failed) { await dlq.add('dead', toDeadPayload(job), { jobId: `dead-${job.queueName}-${job.id}` }); } } } 

The nastier poison pill: jobs that kill the worker

A job that causes an out-of-memory crash or hangs the event loop doesn't throw an error you can catch. The process dies, the job's lock expires, and BullMQ marks it stalled and hands it to another worker, which crashes too. BullMQ limits this with maxStalledCount (default 1): after that many stalls, the job is failed with a "stalled more than allowable limit" error.

const worker = new Worker('emails', processor, { connection, maxStalledCount: 1, // how many times a job may stall before it's failed lockDuration: 30_000, // must exceed your longest event-loop block }); 

Practical guidance:

• Keep maxStalledCount low. One retry after a stall covers a genuine worker crash (a deploy, a pod eviction). More than that mostly multiplies the damage of a real poison pill.
• Alert on stall-failures specifically. A job that failed because it stalled repeatedly probably crashed things. Treat it as higher severity than an ordinary exhausted retry.
• Bound memory per job (stream large inputs rather than loading them whole) and keep processors off the event loop's critical path (use worker threads or a separate process for CPU-heavy work).
• Our graceful-shutdown guidance matters here too, since a worker that stops cleanly on deploy doesn't create stalled jobs at all.

Step 5: Replay deliberately

A DLQ is only useful if you can get jobs back out once you've fixed the cause. Build replay as a reviewed, rate-limited action, not a button that blindly requeues everything.

// replay.ts export async function replayDead(opts: { limit: number; filter?: (d: DeadPayload) => boolean }) { const dead = await dlq.getJobs(['waiting'], 0, opts.limit - 1); let replayed = 0; for (const entry of dead) { const d = entry.data as DeadPayload; if (opts.filter && !opts.filter(d)) continue; await emailQueue.add(d.name, d.data, { // fresh id so BullMQ doesn't treat it as a duplicate of the dead original jobId: `replay-${d.originalJobId}-${Date.now()}`, attempts: 3, // a modest budget: if it fails again, it's still bad }); await entry.remove(); // add first, remove second replayed++; await sleep(50); // gentle: don't re-flood the queue } return replayed; } 

Practices that keep replay safe:

• Add first, remove second. A crash in between means a duplicate replay rather than a lost job, and your handlers are idempotent (they should be) so that's survivable.
• Replay a few first. Pick one or two, confirm they succeed, then do the batch.
• Filter by cause. "Replay everything that failed with ProviderTimeout between 14:00 and 14:20" is the common case after an outage.
• Rate-limit the replay, so thousands of requeued jobs don't recreate the incident that created them.
• Tag replays (replayedFrom) in the job data or logs so you can tell them apart in metrics.
• Fix, then replay. Replaying a poison pill without fixing the code just refills the DLQ.

For payload bugs where the data is wrong, replay can include a transform step (fix the malformed field, then requeue) written as a one-off script, with the original preserved in the DLQ entry until you're done.

Alerting: make the DLQ impossible to ignore

A DLQ nobody watches is a graveyard. Alert on:

• DLQ depth going from 0 to >0, which means something is newly failing terminally
• Rate of arrival (jobs per minute), where a sudden burst means a systemic cause (a bad deploy, a dead dependency)
• Age of the oldest DLQ entry, which catches issues everyone has quietly agreed to ignore
• Retry rate: the ratio of retried attempts to total attempts. A rise means a dependency is degrading before jobs start dying

import client from 'prom-client'; new client.Gauge({ name: 'bullmq_dlq_depth', help: 'Jobs waiting in the dead letter queue', labelNames: ['queue'], async collect() { this.set({ queue: 'emails' }, await dlq.getWaitingCount()); }, }); 

Group alerts by error type so one root cause pages once, not four hundred times. A DLQ entry should also be a ticket-worthy event: every dead job is a bug, a data problem, or a dependency issue that someone should classify.

Failed set vs. explicit DLQ: which should you use?

If you stay with the failed set, set removeOnFail with a generous age and count (never true), and monitor queue.getJobCounts('failed') the same way as above. You lose nothing important, and you skip the reconciler.

Hygiene and security

• Expire DLQ entries deliberately (for example, after 30 days) with a scheduled cleanup, since unbounded retention is the same memory leak we covered in the post about BullMQ memory growth.
• DLQ entries contain payloads. If jobs carry personal data, the DLQ is a second copy of it. Apply the same retention, access control, and redaction rules, and consider storing references (IDs) in job data instead of full records.
• Don't log full payloads in failure messages.
• Make handlers idempotent. Retries, stall recovery, and replays all mean a job can run more than once. Use idempotency keys for external effects.

Test the failure paths

it('retries transient failures with backoff, then dead-letters', async () => { process.env.BACKOFF_BASE_MS = '10'; // fast backoff for tests mailer.send = vi.fn().mockRejectedValue(Object.assign(new Error('boom'), { status: 503 })); const job = await emailQueue.add('welcome', validPayload); await expect(job.waitUntilFinished(queueEvents)).rejects.toThrow(); expect(mailer.send).toHaveBeenCalledTimes(5); // all attempts used await waitFor(async () => (await dlq.getWaitingCount()) === 1); }); it('does not retry permanent failures', async () => { const job = await emailQueue.add('welcome', { userId: 'not-a-uuid' }); await expect(job.waitUntilFinished(queueEvents)).rejects.toThrow(/Invalid payload/); expect(