Processing 5 Million Records with Django and Celery: A Practical Pattern

A simple for loop over Order.objects.all() works fine until it meets 5 million rows. Then you get memory spikes, timeouts, a slow database, and no idea where the job stopped.

Here is the pattern that turns that into a restartable pipeline.

The core idea

Don't run one big job. Run thousands of small, independent Celery tasks, each handling one ID range.

CHUNK_SIZE = 2000 @shared_task def dispatch_order_recalculation(): bounds = Order.objects.aggregate(lo=Min("id"), hi=Max("id")) if bounds["lo"] is None: return 0 start = bounds["lo"] while start <= bounds["hi"]: end = start + CHUNK_SIZE - 1 process_order_chunk.delay(start, end) start = end + 1 @shared_task(bind=True, max_retries=3, acks_late=True) def process_order_chunk(self, start_id, end_id): try: orders = list( Order.objects.filter(id__gte=start_id, id__lte=end_id) .only("id", "subtotal", "tax_rate", "discount") ) for order in orders: order.total = calculate_total(order) Order.objects.bulk_update(orders, ["total"], batch_size=500) except Exception as exc: raise self.retry(exc=exc, countdown=30 * (self.request.retries + 1)) 

Checklist

• Keyset pagination, not OFFSET. Range queries on an indexed column stay fast across the whole table.
• .only() + bulk_update(). Fewer bytes read, far fewer queries. It skips save() and signals, so set updated_at yourself if needed.
• Idempotent tasks. Compute from source data, use upserts, and keep processed flags for emails or payments.
• Celery settings. acks_late=True, prefetch_multiplier=1, time limits, and a dedicated bulk queue.
• Protect Postgres. Limit concurrency, keep one chunk per transaction, index filter columns, and consider PgBouncer.
• Track progress. Use a BulkJob model with F() counters so failed ranges can be requeued alone.

BulkJob.objects.filter(pk=job_id).update(completed_chunks=F("completed_chunks") + 1) 

Chunk size

There's no universal number. Start with 1,000 to 5,000 records, benchmark 500 / 2,000 / 5,000 on your real data, and aim for tasks that take a few seconds to a minute.

Full guide

The complete version, with Celery settings, common mistakes, a production checklist, and an FAQ, is on my blog:

👉 https://blog.reiteachaigean.com/blog/process-5-million-records-django-celery

What's the largest dataset you've processed with Celery? Share your setup below.