The moment you draw a queue on the whiteboard, an interviewer has four follow-up questions ready: what happens if whoever's processing the message crashes halfway through, what happens if the same message gets delivered twice, is the order of messages preserved, and what happens if messages arrive faster than they can be processed. Walk in with all four answers ready, and this becomes one of the strongest parts of your whole design.

Why a queue, and where the line goes

A queue does four genuinely different jobs, and being able to name which one you actually need is what separates "I'd just add a message broker like Kafka" from a design that's actually been thought through.

Here's the line to draw in every design you build: anything the user's response doesn't depend on goes behind a queue. Creating an order has to save that order and reply right away. Sending the confirmation email, updating a recommendation model, indexing for search, and telling the warehouse can all happen a moment later, in the background.

There are real costs here, and you should mention them yourself before you're asked.

• The delay becomes visible to users. "Your order is placed" shows up instantly, but the confirmation email might arrive 30 seconds later, and an analytics dashboard might lag behind by more.
• Debugging gets harder. A failure now shows up in a background worker's log, not in the original request's trace, unless you deliberately carry a shared trace ID through both.
• It's another whole system to run: the message broker itself, how its partitions are managed, watching for messages piling up, and handling messages that keep failing.

Remember: if the user's response doesn't depend on it, put it behind a queue. Everything else stays in the request path.

Queue vs log: the distinction that matters

Picture a to-do list, where you cross an item off and it's gone for good, compared with a diary, where every entry stays on the page forever and you can flip back and reread any day you like. A to-do list works like a message queue. A diary works like what's called a distributed log. Picking the wrong shape for the job is a common and expensive mistake.

A message queue, like RabbitMQ, SQS, or ActiveMQ, delivers each message to exactly one consumer, who processes it and confirms it, and the broker then deletes it. The broker keeps track of each individual message's state.

A distributed log, like Kafka, Kinesis, Redpanda, or Pulsar, appends every message to an ordered, kept log, split into partitions. Consumers each track their own position, called an offset, and read at their own pace; a message is never deleted just because someone read it. Many separate groups of consumers can all read the very same stream, each independently.

The decision rule is simple: do you need to replay history, or have several independent consumers each read the same events? Use a log. Do you need per-message routing, priorities, or delayed retries? Use a queue.

In Kafka, partitions are the unit both of how much work can happen in parallel and of ordering itself: messages sharing the same key always go to the same partition and stay strictly ordered there, and each partition is only ever read by one consumer within a given group. So the number of partitions is the most work you can do in parallel, and the key you choose is what keeps things belonging to the same entity in order; keying by user_id, for instance, keeps one user's own events ordered, even though the topic as a whole is not.

Remember: a queue deletes a message once it's read. A log keeps every message, so many independent readers can each replay it at their own pace. Pick a log whenever you need to replay history.

Delivery semantics and idempotent consumers

There are three possible promises a message system can make, and only two of them are actually achievable.

At-least-once is both the default and the right choice, because losing work outright is usually worse than accidentally doing it twice. That means handling duplicates isn't an edge case; it's a real design requirement.

When Kafka advertises "exactly-once," it means exactly-once inside Kafka itself, through transactional writes across its own topics with a producer that avoids sending the same thing twice. The moment your consumer actually charges a card or sends an email, that's an effect happening out in the real world, one that Kafka's own transaction has no power to undo. The honest way to describe this, and the answer interviewers actually want to hear, is: "at-least-once delivery plus idempotent processing gives you effectively-once behaviour."

Here's how you actually make a consumer idempotent, meaning safe to run more than once:

Two more patterns worth knowing by name.

Conditional writes: a statement like UPDATE orders SET status='shipped' WHERE id=? AND status='paid' has the same effect no matter how many times you run it, because it only applies once the row is actually in the expected starting state.

Version or sequence checks: ignore any incoming event whose version number isn't exactly one greater than the one you already have stored.

Remember: duplicates are guaranteed in a distributed system, so design for them. "At-least-once delivery plus an idempotent consumer" is the honest, achievable promise, not "exactly-once."

Ordering, retries, dead letters, and backpressure

Ordering across an entire topic is expensive, and you almost never actually need it. Ordering per entity usually is what you need instead. Partitioning by the entity's own key, like user_id, account_id, or conversation_id, gives you strict order exactly where it matters, while still letting different entities be processed in parallel. If you genuinely need every single message globally ordered, you're stuck with one partition and one consumer; say that cost out loud when you propose it.

Ordering quietly breaks in three common situations: several consumers sharing one queue, a retried message getting pushed behind newer ones that arrived after it, and changing the number of partitions, which changes which partition a given key lands in.

Retries should use exponential backoff with a bit of randomness added, called jitter. If every failed client retries after exactly the same fixed delay, they all end up retrying together in a wave, called a thundering herd.

Always put a cap on the number of attempts. Also tell apart failures that are worth retrying, like a timeout or a temporary overload, from ones that are permanent, like a validation error; retrying a message that's simply malformed forever is pure waste, and it blocks everything behind it in the same partition.

A dead letter queue is where a message goes after failing too many times. After N attempts, move it there along with its error and how many times it was tried. Nothing is lost, the main flow keeps moving, and a person, or a separate fixed-up consumer, can retry it later. Always alert on how many messages are sitting in the dead letter queue; a dead letter queue nobody is watching is a data-loss incident that nobody has noticed yet.

Backpressure is what happens when producers create messages faster than consumers can process them. The queue keeps growing, memory and disk fill up, and response times climb until something breaks. Your options are:

1. Scale up the number of consumers, automatically, based on how deep the queue is or how far behind consumers are. In Kafka, this only helps up to the number of partitions you have.
2. Put a hard limit on the queue's size, and reject or pause producers once it's full. Failing immediately and clearly is better than slowly grinding to a halt.
3. Shed load on purpose: drop lower-priority messages, or only process a sample of them.
4. Rate limit the producers themselves.

The single most important metric here is consumer lag, meaning how far behind consumers are from the latest message. In Kafka this is the gap between offsets; in SQS it's how old the oldest unprocessed message is. Watch the trend in that lag over time, not just its current value: lag that keeps steadily growing means you simply don't have enough consumers, and waiting longer will never fix that on its own.

Remember: one bad message can block a whole partition if you keep retrying it forever. Move it to a dead-letter queue after N attempts, and alert on that queue's depth.

Event-driven patterns: pub/sub, outbox, event sourcing, CQRS

Point-to-point versus publish/subscribe. Point-to-point means one producer, one consumer, one queue, like a personal to-do list. Publish/subscribe, often shortened to pub/sub, means one event has many independent subscribers, like a public announcement everyone can hear. Prefer publishing facts, like "order placed," over issuing commands, like "send the email." Publishing a fact lets you add a fourth listener later without ever touching the original producer, which is the entire point of decoupling systems in the first place.

The dual-write problem, and the outbox pattern that fixes it, is the single most valuable idea in this whole topic. Picture writing a cheque, and then, on a separate trip, walking to the post office to mail someone a receipt for it. If you get hit by a bus in between those two trips, the cheque now exists, but nobody was ever told about it. Saving something to a database and then separately publishing an event to a message broker has exactly this same problem: it isn't one single atomic action. A crash in between the two can leave the order saved but never announced, or announced without ever actually being saved.

The reverse of this idea, called the inbox pattern, removes duplicates on the receiving side, by recording each processed message's id in the very same transaction as the effect it caused.

Event sourcing stores the full sequence of events as the actual source of truth, and figures out the current state by replaying them from the start. In exchange, you get a complete audit trail, the ability to look at any point in the past, and the option to build entirely new views from that same history later. The cost is having to handle old event formats as your system evolves, needing periodic snapshots so replaying isn't unbounded, and the fact that a simple question like "what's the current balance" now requires computation instead of a lookup. Use this where the history genuinely is the product, like ledgers, audit trails, or collaborative documents, not as a default choice everywhere.

CQRS, short for command query responsibility segregation, splits the model you write to from one or more separate models you read from, kept in sync through events. It pairs naturally with event sourcing, and with any system whose reads look very different in shape from its writes, such as a normalised write side feeding a denormalised feed for reading. The cost is that the two sides can briefly disagree with each other, and you now maintain two models instead of one.

Change data capture, often shortened to CDC, reads a database's own internal change log directly, using a tool like Debezium on Postgres's write-ahead log or MySQL's binary log, and publishes each row change as an event. This gives you the outbox pattern's same safety without changing your application code at all, and it's the standard way to feed search indexes, caches, and data warehouses from one authoritative database.

Remember: writing to a database and publishing an event are two separate actions, so a crash between them loses one or the other. The outbox pattern writes the event into the same database transaction, so there is nothing to lose.

Quick recap