A job queue is the system behind "process this in the background": Sidekiq, Celery, SQS, and every button in a real product that says "we'll email you when it's ready." A producer enqueues a job, a worker picks it up later and runs it. This question tests whether you understand delivery guarantees, retry logic, and what happens to ordering once you have more than one worker.

Requirements

Functional requirements

• Producers enqueue jobs with a payload, like "resize image X" or "send email Y."
• Workers pull jobs, run them, and report success or failure.
• Failed jobs retry with backoff, up to a maximum number of attempts.
• Jobs can be scheduled to run in the future.
• Jobs support priorities: high, default, low.
• Jobs that fail permanently land in a dead letter queue for a human to inspect.

Non-functional requirements

• At-least-once delivery: never silently drop a job.
• Thousands of workers can pull from the same queues at once.
• Enqueueing must be fast; a producer should never block waiting on a job to run.
• Someone can see queue depth, failure rate, and inspect a stuck job.
• Job handlers must be idempotent, since at-least-once delivery means a job can run twice.

Remember: "at-least-once" is a promise about delivery, not about your handler's side effects. Making the handler safe to run twice is your job, not the queue's.

Estimates

Assume a mid-size product with 50 million jobs a day.

• Average rate: 50,000,000 / 86,400 ≈ 580 jobs/sec.
• Peak (3x during business hours): roughly 1,750 jobs/sec.
• Storage: at 1 KB per job and a 7-day retention window for debugging, that's about 350 GB, comfortable for a single Postgres instance or a Kafka topic with retention.
• Worker fleet: if a job takes 200ms on average, one worker handles 5 jobs/sec. Sustaining 1,750 jobs/sec needs about 350 concurrent workers; provision 500+ for headroom.

Say this arithmetic out loud in an interview. It's what justifies picking a DB-backed queue over Kafka, or the other way around, later.

API

Workers never call this HTTP API to fetch jobs. They speak the broker's native pull protocol directly: Redis BLPOP, SQS ReceiveMessage, or a database SELECT ... FOR UPDATE SKIP LOCKED. The HTTP surface above exists for producers and for humans operating the system.

Data model

If you use a broker like Redis, SQS, or Kafka instead of a DB-backed queue, this table becomes the job's status and audit record in Postgres, while the broker holds the lightweight queue of message pointers. Many real systems, including Sidekiq and SQS-backed setups, run this exact hybrid: a fast broker for delivery, a durable database for state.

High-level design

Three moving pieces do three different jobs: the broker delivers, the metadata store tracks state and audit history, and workers execute. Some systems collapse the broker and metadata store into one database-backed queue, simpler to run but with a lower throughput ceiling. Others split them, Kafka for delivery and Postgres for state, trading two systems to keep in sync for a much higher ceiling.

Deep dives

Why is leasing better than locking a row forever?

A worker doesn't hold a database lock for the entire time it processes a job. That would block visibility into the job and make crash recovery painful. Instead it uses a visibility timeout: UPDATE jobs SET status='in_progress', locked_by=$1, locked_at=now() WHERE id=$2 AND status='queued'. If the worker crashes mid-job, a reaper process periodically finds jobs whose locked_at is older than the visibility timeout and requeues them. SQS calls this a visibility timeout; Sidekiq calls it a watchdog. Same idea, different name.

Remember: a job stuck at "in_progress" past its visibility window isn't stuck, it's waiting for the reaper to notice and hand it to someone else.

How does retry and the dead letter queue work?

Retry with exponential backoff plus jitter: delay = base * 2^attempts + random_jitter, capped at something reasonable like 15 minutes. Jitter matters because without it, every failed job retries at the exact same instant and can re-trigger the very outage it's recovering from.

After max_attempts, move the job to status = 'dead' and stop retrying automatically. A human or automated process inspects dead jobs, fixes the root cause, and requeues them through POST /dlq/{id}/requeue. Never retry forever: a permanently broken payload retried endlessly wastes capacity and can hide a real outage inside your dashboards.

How do priorities and ordering actually work?

The cheapest way to implement priority is separate queues per level (queue:high, queue:default, queue:low), with workers polling high before default before low, using weighted round-robin so low-priority jobs don't starve completely. A single queue with a priority sort column gets expensive to sort on every dequeue under real load.

Full global ordering across a sharded, multi-worker queue is expensive and rarely needed. If one specific ordering guarantee matters (say, "events for the same user must process in order"), use a partition key: Kafka partitions, or pinning a key to one worker, so everything for that key lands on the same worker in order while different keys still run in parallel.

Trade-offs and follow-up questions

Pick based on scale: under roughly 1,000 jobs/sec, a DB-backed queue or Redis is the pragmatic choice. Above that, or if you need to replay events at the stream level, reach for Kafka.

Q: How do you guarantee exactly-once processing when the network can duplicate delivery? A: You don't guarantee it at the delivery layer, you guarantee it at the application layer with an idempotency key. The handler checks a dedup row (INSERT ... ON CONFLICT DO NOTHING, keyed on a deterministic key derived from the job) before doing the actual work. Delivery stays at-least-once; the effect becomes exactly-once.

Q: A worker crashes mid-job. What happens to that job? A: It stays in_progress with a locked_at timestamp. A reaper scans for jobs whose lease has expired and requeues them, incrementing attempts and resetting status. This is exactly why handlers must be idempotent, since the job may partially run twice.

Q: How do you stop one noisy queue from starving the others? A: Separate queues per tenant or job type, with weighted round-robin polling or dedicated worker pools per queue, plus rate-limiting on the producer side if one source is flooding the system.