BullMQ in NestJS: Retries, Idempotency and Dead-Letter Queues

A queue that retries blindly is worse than no queue at all. It double-sends the welcome email, double-charges the customer, and then deletes the evidence by retrying the same broken job until the failure looks like noise. This post builds the setup I actually run in NestJS: retries that back off with jitter, jobs that cannot run twice, and a dead-letter queue that keeps every terminal failure inspectable and replayable.

I learned this the way most backend lessons arrive: a welcome email went out three times. The endpoint that sent it called the mail provider directly, the provider timed out once, and my retry logic ran the whole send again instead of checking whether the first attempt had landed. The user got three identical emails, support got a ticket, and I got a rule I have kept since: a background job is only safe to retry if running it twice is harmless, and a job that fails for good must land somewhere a human can find it.

That is what this post builds, in one NestJS module. The stack is BullMQ 6.7.0 with @nestjs/bullmq 12.0.0 on NestJS 11, verified against the installed packages on November 5, 2026: every snippet below typechecks under strict TypeScript, and the backoff numbers are measured from BullMQ's own code, not copied from a tutorial. It is the same NestJS and Redis pair I ship in production, the one behind the real-time rooms in the Healthcare Ireland build that I wrote up in Scaling Socket.io Rooms in NestJS With the Redis Adapter and the hiring page at Hiring a Senior Next.js + NestJS Developer from Pakistan.

The three failures a naive queue hides

A queue with default settings hides three failures at once. It retries instantly, so a struggling dependency gets hammered by every failed job at the same moment. It repeats side effects, so a job that crashed after sending the email sends it again on the retry. And it deletes its own evidence, because a failed job that is cleaned up too aggressively disappears before anyone reads the error. Each failure is cheap to prevent and expensive to debug after the fact, and all three come from treating the queue as a fire-and-forget box instead of a part of the system with its own failure modes.

The fix is three mechanisms that belong together: backoff with jitter for retries, idempotency at two layers for repeated runs, and a dead-letter queue for terminal failures. None of them is optional if the job touches money, email, or any external system that charges per call. Skip one and the other two quietly stop protecting you: backoff without idempotency just spaces out the duplicate sends, and idempotency without a dead-letter queue means a permanently broken job retries five times and then vanishes.

Set up the queue the documented way

Install two packages and register one module. BullMQ is the queue library; @nestjs/bullmq is the NestJS wrapper around it. The NestJS docs are direct about which one to pick in 2026: the older Bull package is in maintenance mode and only gets bug fixes, while BullMQ is actively developed. If you are starting now, BullMQ is the choice, and the wrapper is the documented way to use it inside NestJS.

npm install --save @nestjs/bullmq bullmq 

The module sets the defaults every job in the queue inherits. Three lines in it deserve attention before you copy the rest.

// notifications.module.ts import { Module } from '@nestjs/common'; import { BullModule } from '@nestjs/bullmq'; import { DeadLetterListener } from './dead-letter.listener'; import { DeadLetterService } from './dead-letter.service'; import { MailService } from './mail.service'; import { NotificationsProcessor } from './notifications.processor'; import { NotificationsService } from './notifications.service'; @Module({ imports: [ BullModule.forRoot({ connection: { host: 'localhost', port: 6379, // Workers block on Redis; without this, ioredis aborts the // blocked command after a few retries during a Redis blip. maxRetriesPerRequest: null, }, }), BullModule.registerQueue({ name: 'notifications', defaultJobOptions: { attempts: 5, // Custom type: BullMQ calls our backoffStrategy for this name. // delay: -1 is the documented placeholder for custom strategies. backoff: { type: 'capped-jitter', delay: -1 }, removeOnComplete: 1000, // Keep failures 30 days for inspection, cap the set at 5000. removeOnFail: { age: 2_592_000, count: 5000 }, }, }), BullModule.registerQueue({ name: 'dead-letter' }), ], providers: [ MailService, NotificationsService, NotificationsProcessor, DeadLetterListener, DeadLetterService, ], }) export class NotificationsModule {} 

maxRetriesPerRequest: null is the line most tutorials skip and the one that bites first in production. BullMQ workers wait for jobs with blocking Redis commands, and the ioredis default aborts a command after a few retries. During a short Redis blip, that abort kills the blocked wait instead of letting the worker ride it out. Setting it to null removes the cap. It belongs in the connection options of every queue you run.

The removeOnFail shape matters as much as the values. Never set it to true on a queue whose failures you need to read: true deletes a failed job the moment it lands in the failed set, which is exactly when you want to look at it. The object form keeps up to 5,000 failed jobs for up to 30 days. One honest caveat from BullMQ's own type definitions: eviction is best-effort, evaluated when another job finishes, so treat the age as a rough bound, not a timer.

Read the Redis URL from the environment in your real app instead of hardcoding localhost. The snippet above uses localhost so you can run it against a local Redis; the shape is what matters, not the host.

Retries: attempts counts total runs, not retries

attempts: 5 means the job runs at most five times: the first run plus up to four retries. The default is 1, which means no retry at all. This is the single most misread option in BullMQ, and it is written plainly in the type definitions, so there is no excuse for getting it wrong once you have read them. If you want three retries after the first failure, set attempts: 4.

The second most misread behavior is what happens without a backoff: BullMQ retries a failed job with no delay, as soon as it fails. Five instant retries from every failed job, hitting a dependency that is already struggling, is how a small outage becomes a bigger one. Backoff is what turns retries from a hammer into a waiting room.

The built-in strategies are enough for most queues. Fixed waits the same time before every retry. Exponential doubles the gap each time. I measured the exponential formula against BullMQ 6.7.0's own code: the delay is 2^(attemptsMade - 1) * delay, so with delay: 1000 the gaps are 1s, 2s, 4s, then 8s. Add jitter: 0.3 and each step is randomized within 70 to 100 percent of the computed delay, which stops a thousand jobs that failed in the same second from retrying in the same second.

When the built-in curve is not enough, write your own. A custom strategy is a function BullMQ calls with the number of attempts already made, and it returns the delay in milliseconds. Returning -1 fails the job immediately instead of retrying. Mine uses full jitter, a random delay anywhere from zero up to an exponentially growing ceiling, capped at 60 seconds, because full jitter spreads retries more evenly than the built-in partial jitter when a whole queue fails at once.

// backoff.strategy.ts /** * Full-jitter exponential backoff with a 60s ceiling. * BullMQ calls this with the number of attempts already made. * Returning -1 would fail the job immediately; we always return a delay. */ export function cappedJitterBackoff(attemptsMade: number): number { const ceiling = Math.min(1000 * 2 ** attemptsMade, 60_000); return Math.floor(Math.random() * ceiling); } 

The strategy is registered on the worker, not the queue, through the second argument of @Processor. One detail that trips people up: BullMQ only calls your function when the job's backoff type is a name it does not recognize as built-in. That is why the module above sets backoff: { type: 'capped-jitter', delay: -1 } as the default: capped-jitter is not a built-in type, so BullMQ hands the decision to the function, and delay: -1 is the documented placeholder for custom strategies.

// notifications.processor.ts (worker options only; the full file is below) @Processor('notifications', { concurrency: 5, limiter: { max: 50, duration: 1000 }, settings: { backoffStrategy: cappedJitterBackoff }, }) export class NotificationsProcessor extends WorkerHost { /* ... */ } 

If you do not need a custom strategy, do not write one. { type: 'exponential', delay: 1000, jitter: 0.3 } in the queue defaults gets you nearly the same behavior with zero code, and less code in the retry path is a feature.

Classify the error or every retry is wasted

Not every failure deserves a retry. A timeout or a 500 from the mail provider might clear in seconds; an invalid email address will fail the same way on attempt five as it did on attempt one. Retrying the second kind burns five attempts, five backoff waits, and five log lines to learn nothing. The fix is to classify the error inside the processor and fail fast on the permanent kind.

BullMQ gives you UnrecoverableError for exactly this. Throw it and the job moves straight to the failed set, skipping whatever attempts remain. Throw a normal error and the job retries with backoff. Throw RateLimitError when a provider rate-limits you and the job goes back to waiting without burning an attempt at all, because rate limiting is not counted as a real failure. One rule from the BullMQ docs applies to all three: thrown exceptions must be real Error objects, not strings or plain objects, or BullMQ cannot handle them correctly.

// notifications.processor.ts import { Processor, WorkerHost } from '@nestjs/bullmq'; import { Job, UnrecoverableError } from 'bullmq'; import { cappedJitterBackoff } from './backoff.strategy'; import { MailService, MailValidationError } from './mail.service'; import { NotificationData } from './notifications.service'; export type NotificationJobName = 'send-welcome-email' | 'send-invoice-email'; @Processor('notifications', { concurrency: 5, limiter: { max: 50, duration: 1000 }, settings: { backoffStrategy: cappedJitterBackoff }, }) export class NotificationsProcessor extends WorkerHost { constructor(private readonly mail: MailService) { super(); } async process(job: Job<NotificationData, void, NotificationJobName>): Promise<void> { switch (job.name) { case 'send-welcome-email': await this.sendWithClassification(job, 'welcome'); return; case 'send-invoice-email': await this.sendWithClassification(job, 'invoice'); return; default: throw new UnrecoverableError(`Unknown job name: ${job.name}`); } } private async sendWithClassification( job: Job<NotificationData, void, NotificationJobName>, template: string, ): Promise<void> { try { await this.mail.send({ to: job.data.email, template, // Same key on every retry: the provider dedupes the send. idempotencyKey: `${job.name}-${job.id}`, }); } catch (err) { if (err instanceof MailValidationError) { // Permanent failure: skip the remaining attempts, go straight to failed. throw new UnrecoverableError(`Invalid address for job ${job.id}: ${err.message}`); } throw err; } } } 

Two NestJS details in that file are worth naming. First, BullMQ has no @Process('job-name') decorator: the older Bull package had one, BullMQ deliberately dropped it, and the NestJS docs tell you to switch on job.name instead. Second, the processor is a class that extends WorkerHost and implements process(), registered as a provider; there is no separate worker bootstrap to write.

The classification rule I use in practice: if the error message names something about the data (validation, not found, unauthorized, bad request), it is permanent. If it names something about the moment (timeout, connection reset, 429, 502, 503), it is transient. When I genuinely cannot tell, I retry: a wasted retry is cheaper than a dropped email, and the dead-letter queue below catches the job if I was wrong.

Idempotency: two layers, and only one of them is optional

Retries assume it is safe to run the job again. That assumption breaks the moment a job has a side effect. A worker can crash after the mail provider accepted the send but before the worker recorded success, and the retry will send the same email a second time. The protection has two layers, and they solve different halves of the problem.

Layer one is a deterministic job id at enqueue time. When you add a job with a jobId that already exists, BullMQ ignores the duplicate add. A double-clicked submit button, a retried HTTP request, or a webhook provider delivering the same event twice all collapse into one job.

// notifications.service.ts import { Injectable } from '@nestjs/common'; import { InjectQueue } from '@nestjs/bullmq'; import { Queue } from 'bullmq'; export interface NotificationData { userId: number; email: string; invoiceId?: string; } @Injectable() export class NotificationsService { constructor(@InjectQueue('notifications') private readonly queue: Queue) {} async sendWelcomeEmail(userId: number, email: string): Promise<string> { const job = await this.queue.add( 'send-welcome-email', { userId, email } satisfies NotificationData, // Deterministic id: a retried HTTP request or a double-click // enqueues exactly one job. No ':' allowed in custom ids. { jobId: `welcome-email-${userId}` }, ); return job.id as string; } async sendInvoiceEmail(userId: number, email: string, invoiceId: string): Promise<string> { const job = await this.queue.add( 'send-invoice-email', { userId, email, invoiceId } satisfies NotificationData, { jobId: `invoice-email-${invoiceId}` }, ); return job.id as string; } } 

Custom ids have two rules, both enforced by BullMQ's own code: they cannot contain a colon, and they cannot be plain integers. Use a readable prefix plus your business id, joined with a hyphen, and you stay clear of both. Note the limit of this layer, because it matters: once a job is removed from Redis, the same id can be added again. removeOnComplete: 1000 in the module above means a completed job's id becomes reusable once it ages out of the completed set. Layer one stops duplicates while the job exists; it says nothing about a job that already finished.

Layer two is the one that actually makes retries safe, and it lives at the provider, not in the queue. Pass the same idempotency key to the mail provider on every attempt of the same job, and the provider dedupes the send at the source. The processor above builds the key from the job name and the job id, so attempt one and attempt four carry the identical key. If your provider does not support idempotency keys, the honest fallback is a unique constraint in your own database: an upsert on the same key, or a sent-log table with a unique index, checked and written in the same transaction as the effect. The naive version, a check-then-set flag in Redis or memory, has a race: the worker can crash after sending and before setting the flag, and the retry sends again. Put the key where the effect happens, not next to it.

One more rule for job data: carry ids, not records. Job data must be serializable, and dead-letter entries keep a copy of the payload, so a job carrying a full user record quietly duplicates personal data into a second store. { userId, email } is enough; the worker fetches the rest.

Dead-letter queue: BullMQ ships the parts, not the queue

BullMQ has no built-in dead-letter queue. It keeps terminally failed jobs in a failed set, and that set is a legitimate dead-letter store on its own. I build an explicit queue anyway, because it gives me separate retention, separate alerting, and a replay path that does not touch the main queue's cleanup settings. The pattern is a listener on the failed event that copies the job into a second queue no worker consumes.

The event semantics are the part most write-ups get wrong, so I verified them against BullMQ 6.7.0's source instead of repeating them. There are two failed events and they fire at different times. The worker-level event fires on every failed attempt, including ones that will be retried. The QueueEvents failed event fires only when the job actually lands in the failed set, which happens when retries are exhausted or UnrecoverableError was thrown. The Lua script that moves a job to finished status emits the failed event and a retries-exhausted event together, only on that terminal path. So a QueueEvents listener sees exactly the jobs that are done failing, and nothing else. I still check getState() === 'failed' inside the listener as a guard; it costs one Redis read and it covers every terminal path, including jobs that stalled too many times.

// dead-letter.listener.ts import { InjectQueue, OnQueueEvent, QueueEventsHost, QueueEventsListener } from '@nestjs/bullmq'; import { Queue } from 'bullmq'; import { NotificationData } from './notifications.service'; export interface DeadLetterData { originalQueue: string; originalJobId: string; name: string; data: NotificationData; failedReason: string; attemptsMade: number; failedAt: string; } @QueueEventsListener('notifications') export class DeadLetterListener extends QueueEventsHost { constructor( @InjectQueue('notifications') private readonly notificationsQueue: Queue, @InjectQueue('dead-letter') private readonly dlqQueue: Queue, ) { super(); } // QueueEvents 'failed' fires only when a job lands in the failed set, // i.e. retries are exhausted or UnrecoverableError was thrown. // (The worker-level 'failed' fires on every attempt instead.) @OnQueueEvent('failed') async onFailed({ jobId, failedReason }: { jobId: string; failedReason: string }): Promise<void> { const job = await this.notificationsQueue.getJob(jobId); if (!job || !job.id || (await job.getState()) !== 'failed') return; await this.dlqQueue.add( 'dead-letter', { originalQueue: 'notifications', originalJobId: job.id, name: job.name, data: job.data as NotificationData, failedReason, attemptsMade: job.attemptsMade, failedAt: new Date().toISOString(), } satisfies DeadLetterData, // Deterministic DLQ id: a duplicate event cannot double-park the job. { jobId: `dlq-${job.id}` }, ); } } 

The listener is registered as a provider, like the processor, and it uses the same queue names the module registered. Keep the full context in the dead-letter entry: original payload, error, attempt count, and timestamps. A dead-letter entry you cannot diagnose is just a bigger problem than the failed job was.

Replay is a method, not a manual Redis edit. It removes the old failed job first, because re-adding the same job id while the old job still exists is silently ignored by BullMQ, then re-adds the job with its original id and removes the dead-letter entry.

// dead-letter.service.ts import { Injectable } from '@nestjs/common'; import { InjectQueue } from '@nestjs/bullmq'; import { Queue } from 'bullmq'; import { DeadLetterData } from './dead-letter.listener'; @Injectable() export class DeadLetterService { constructor( @InjectQueue('notifications') private readonly notificationsQueue: Queue, @InjectQueue('dead-letter') private readonly dlqQueue: Queue, ) {} async replay(dlqJobId: string): Promise<'replayed' | 'not-found'> { const dlqJob = await this.dlqQueue.getJob(dlqJobId); if (!dlqJob) return 'not-found'; const { originalJobId, name, data } = dlqJob.data as DeadLetterData; const failedJob = await this.notificationsQueue.getJob(originalJobId); // Remove the failed original first: re-adding the same jobId while // the old job still exists is silently ignored by BullMQ. if (failedJob) await failedJob.remove(); await this.notificationsQueue.add(name, data, { jobId: originalJobId }); await dlqJob.remove(); return 'replayed'; } async ...