Picture a waiter who takes your order, then stands at your table while the kitchen cooks it, blocking every other customer from being served. That is a web server that does slow work inside the request itself: sending an email, generating a report, running a model. Once one request takes 10 seconds, everything behind it queues up too. A background worker is the fix: the waiter drops your order in the kitchen and immediately moves to the next table.

Celery is the standard Python answer to this problem, and interviews push past "I used @shared_task" into the two things that actually break in production: a task running twice, and a worker dying mid-task. This lesson builds both answers with real mechanisms, then covers routing, chains, and how Celery compares to a couple of common alternatives.

Celery's moving pieces

Celery has three parts:

• Broker (Redis or RabbitMQ): a message queue holding tasks waiting to run. Your app pushes a task message here; it never runs the task itself.
• Worker(s): separate processes that pull messages off the broker and execute the task function. You can run many workers, on many machines, all consuming from the same queue.
• Result backend (optional: Redis, a database): stores a task's return value and state, if something later needs to check on it.

The web app never talks to a worker directly. It serializes the task's name and arguments into a message and publishes it to the broker. Some worker, possibly on a different machine, picks it up whenever it's free. That decoupling is the entire point: the request returns immediately, and the real work scales independently of your web tier.

Remember: the broker holds pending work, the worker runs it, and the result backend (if configured) remembers what happened. The web app only ever talks to the broker.

Retry with backoff, and why jitter matters

retry_jitter is the detail interviewers listen for. Without it, if 1,000 tasks fail at the same moment, say a downstream service blips, they all retry at exactly the same backed-off intervals, producing a synchronized retry storm that can re-crash the very service they're retrying against. Jitter spreads that storm out over time instead.

For cases needing custom logic instead of autoretry_for:

Remember: add jitter to every backoff. Without it, a thousand tasks failing together retry together, turning a blip into a second, synchronized outage.

Idempotency: the number one Celery interview question

The premise behind "how do you ensure idempotency" is that Celery's default guarantee is at-least-once delivery, not exactly-once. A worker can execute a task, crash before acknowledging it to the broker, and the broker redelivers it to another worker: now it has run twice.

The fix is making the task's effect idempotent, not trying to prevent redelivery, which you can't reliably do:

Two standard mechanisms: an idempotency key plus a dedup store (shown above, checked before the side effect happens), or natural idempotency, designing the operation so running it twice has the same result as running it once. UPDATE orders SET status = 'shipped' WHERE id = %s is naturally idempotent. INSERT INTO shipments ... is not, unless you add a unique constraint on order_id and catch the conflict.

Remember: you can't reliably stop a redelivery, so the dedup check happens before the side effect, using a key generated once by the caller, not once per retry.

What happens when a worker dies

This depends entirely on acknowledgment mode.

• Early ack (the default): the worker acks the message as soon as it starts the task. If the worker dies mid-task, the message is already gone from the broker: the task is silently lost. Fine for best-effort, non-critical work.
• Late ack (task_acks_late=True): the worker acks only after the task finishes. If the worker crashes mid-task, the broker never got an ack, so it redelivers the message to another worker. Safer for critical work, but it means a task that crashes the worker itself will be retried, which is exactly why the task body must be idempotent.

worker_prefetch_multiplier = 1 matters alongside late-ack: with the default prefetch, a worker reserves several tasks at once, and if it crashes, all of those reserved-but-unstarted tasks get redelivered together, potentially all landing on one already-busy worker.

Remember: early ack can silently lose a task on worker crash; late ack redelivers it but requires the task to be idempotent. Pick late ack plus prefetch_multiplier=1 for anything that must actually complete.

Task routing and priority queues

By default every task goes to one queue and any worker can pick it up. Routing sends different tasks to different queues, so a dedicated pool of workers can handle slow tasks (video processing) separately from fast ones (sending an email). Otherwise, one slow task blocks a worker that could have cleared ten fast ones in the meantime.

Celery has no true priority within a queue on Redis by default; it's FIFO. To get priority, either run separate queues per priority tier, as shown above, or switch to RabbitMQ as the broker, which supports a real per-message priority field. The interview answer: "priority" in Celery is almost always implemented as queue separation, not a priority field, because the default broker doesn't support message priority at all.

Chains run tasks sequentially, each one receiving the previous task's return value. Chords run a group of tasks in parallel, then fire a callback once every one of them finishes, with their results collected as a list.

A chord's callback is itself just another task. Celery tracks completion with a counter in the result backend, decrementing it as each group member finishes and firing the callback at zero. This is why a chord requires a result backend: without one, there is no way to know when the group is actually done.

Remember: "priority" in Celery usually means separate queues, not a priority field, because the default broker (Redis) doesn't support one. A chord needs a result backend to know when its group is finished.

Custom task states, periodic tasks, and the result backend

Beyond Celery's built-in states, update_state reports custom progress, essential for a long task a frontend needs to show progress for.

The result backend (commonly Redis or Postgres) is where Celery writes task state and return values, separate from the broker, which only queues the task messages themselves. Without a result backend, task.delay() still runs the task, but AsyncResult.status stays PENDING forever and .get() hangs, because there's nowhere for the answer to be written.

Periodic tasks run through celery beat, a separate scheduler process that pushes tasks onto the broker at configured times; actual execution still goes through normal workers.

Run exactly one beat process. Two of them will double-schedule everything.

Remember: the broker queues messages; the result backend stores outcomes. They're conceptually separate even when both point at the same Redis instance, and only one of them tells you whether a task is done.

Choosing between Celery, Temporal, and a custom worker

Celery is not the only option, and knowing when to reach for something else is itself an interview-worthy answer.

Temporal is a workflow orchestration engine. You write a workflow function that calls "activities," the actual side-effecting steps, and Temporal's server durably persists execution state after every step. If a worker crashes mid-workflow, a new worker resumes from the last completed activity, not from the start. That's the specific thing Celery doesn't give you: durable, resumable, multi-step state, plus a UI showing exactly which step a given execution is on. The cost is real: Temporal needs its own server plus Postgres plus a UI, roughly 6-7x heavier at idle than a Redis-plus-worker Celery setup, which matters on a small cluster.

A custom DB-backed worker claims rows from a jobs table using SELECT ... FOR UPDATE SKIP LOCKED, so multiple worker processes never double-process the same row:

skip_locked=True is the key detail: a second worker running the same loop skips rows this one already locked, instead of blocking on them. This option needs zero new infrastructure if you already run Postgres, at the cost of hand-rolled retry logic and a throughput ceiling in the low thousands of jobs per second.

The decision framework: simple, independent tasks (send an email, resize an image) or a small, resource-constrained cluster both point at Celery. A multi-step pipeline needing step-by-step visibility and resumability across a crash, parse, chunk, run AI calls, generate output, points at Temporal. A team that wants low overhead now but may grow into complex workflows later should start with Celery and migrate only the specific workflow that outgrows it, rather than moving the whole system upfront.

Remember: Celery for simple or independent tasks; Temporal for multi-step workflows that must resume from the last completed step after a crash. Start simple and migrate the one workflow that outgrows it, not the whole system.

BullMQ: the same idea in Node

BullMQ is a Redis-backed job queue for Node.js, the JavaScript-ecosystem equivalent of Celery. The core idea is identical: move slow work out of the request/response cycle onto workers pulling from a shared queue.

emailQueue.add pushes the job onto a Redis-backed structure and returns immediately, without waiting for the email to send. A separate Worker process, possibly on a different machine, picks it up, runs the callback, and if it throws, BullMQ retries automatically using the configured exponential backoff, up to the attempts limit. What BullMQ adds over a plain Redis list: delayed jobs, automatic retries with backoff, priority queues, and per-queue rate limiting, the same category of feature Celery gives Python. A candidate coming from Python can map BullMQ concepts onto Celery concepts almost one to one: Queue.add is task.delay(), attempts/backoff is autoretry_for/retry_backoff, and a Worker is a Celery worker process.

Remember: BullMQ is Celery's Node equivalent: a Redis-backed queue, a worker pool, and built-in retries with backoff. The concepts transfer almost directly between the two ecosystems.