Chapter 17 of 33

Message Queues & Pub/Sub

When a customer places an order on a shopping site, a lot has to happen: save the order, charge the payment, send a confirmation email, send an SMS, tell the warehouse, update analytics, and award loyalty points. If the order service calls each of these one after another and waits for every reply, the customer stares at a spinner, and if the SMS provider is slow or down, the whole order fails.

Most of those steps do not need to finish before the customer sees "Order placed". They just need to happen reliably, soon. The tool for "do this later, but definitely do it" is a message queue, and the tool for "tell everyone who cares that this happened" is publish/subscribe (pub/sub).

Synchronous (tightly coupled)                Asynchronous (decoupled)

Order ──→ Payment                             Order ──→ Payment         (still sync: must succeed)
  │ ──→ Email   (wait)                          │
  │ ──→ SMS     (wait, slow!)                   └──→ [ broker: "OrderPlaced" ]
  │ ──→ Warehouse (wait)                                 ├──→ Email service
  └ ──→ Analytics (wait)                                 ├──→ SMS service
                                                         ├──→ Warehouse service
response time = sum of all calls                         └──→ Analytics service
one failure fails the order                   response time = save + pay + publish

This chapter explains how queues and pub/sub work, what delivery guarantees they really give, how ordering works, and how Kafka, RabbitMQ and Amazon SQS differ.


Core Vocabulary

A message queue is a component that stores messages sent by producers until consumers receive and process them, so the sender and receiver do not need to be running, fast, or available at the same time.

  • Message: a small piece of data, usually JSON, Avro or Protobuf, describing a task ("resize image 42") or an event ("order 101 placed"). Plus metadata such as an ID, timestamp and key.

  • Producer (publisher): the service that sends messages.

  • Consumer (subscriber): the service that receives and processes them.

  • Broker: the server (cluster) that stores and delivers messages: Kafka, RabbitMQ, SQS, Google Pub/Sub and so on.

  • Acknowledgement (ack): the consumer's signal "I finished this message; you can forget it".

Why put a broker in the middle?

Benefit

What it means

Example

Decoupling

The producer does not know who consumes, or how many consumers exist.

Add a fraud-check consumer without touching the order service.

Asynchronous processing

The user gets a fast response; slow work happens in the background.

Video upload returns at once; transcoding runs later.

Load levelling

The queue absorbs spikes; consumers work at a steady rate.

Result-day traffic is 20× normal; the PDF generator still runs at its own pace.

Fault tolerance

If a consumer is down, messages wait instead of being lost.

The SMS gateway is down for 10 minutes; SMSs go out when it returns.

Easy scaling

Add more consumer instances to drain the queue faster.

Run 50 thumbnail workers during a sale, 5 at night.


Two Messaging Patterns

1. Point-to-point (work queue)

Each message is processed by exactly one of the consumers. Several identical workers read from the same queue and share the work; this is called the competing consumers pattern.

                         ┌──→ Worker 1   gets m1, m4
Producer ──→ [ m1 m2 m3 m4 m5 m6 ] ──┼──→ Worker 2   gets m2, m5
                         └──→ Worker 3   gets m3, m6

Use it for tasks: send this email, resize this image, generate this report. You want each task done once, by whoever is free.

2. Publish/subscribe (fan-out)

A message published to a topic is delivered to every subscriber of that topic. Each subscriber gets its own copy.

                              ┌──→ Email service      (gets every event)
Order service ──→ topic "orders" ──┼──→ Warehouse service  (gets every event)
                              └──→ Analytics service  (gets every event)

Use it for events: "OrderPlaced", "UserSignedUp", "PaymentFailed". The publisher announces a fact; any number of services react in their own way.

Combining them

In practice you combine both: each subscribing service gets every event (pub/sub), and inside that service several instances share the load (competing consumers). On AWS this is the classic SNS topic → one SQS queue per service fan-out. In Kafka it is built in through consumer groups, as we will see below.

Aspect

Point-to-point queue

Pub/sub topic

Who gets a message?

One consumer

Every subscriber

Message represents

A task / command

An event / fact

Adding a consumer

Shares the existing work

Adds a new copy of the stream

Examples

SQS queue, RabbitMQ queue, Celery tasks

SNS, Google Pub/Sub, Kafka topic, RabbitMQ fanout exchange


Delivery Guarantees

Networks drop packets and processes crash at the worst moment. A consumer might process a message and then crash just before sending the ack. Did the work happen or not? The broker cannot know. Every messaging system therefore picks one of three guarantees.

Guarantee

How it works

Risk

Good for

At-most-once

Ack (or commit) before processing; never retry.

Messages can be lost.

Metrics samples, live location pings

At-least-once

Ack after processing; retry if no ack arrives.

Messages can be duplicated.

The default for almost everything

Exactly-once

At-least-once delivery + deduplication / transactions.

More complex and slower; only within defined boundaries.

Payments, billing, counters

In practice, "exactly-once" means exactly-once effect: messages may be delivered more than once, but processing them is idempotent, so the result is as if each was processed once.

Visibility timeout, retries and dead-letter queues

Here is how a typical queue (Amazon SQS works like this) implements at-least-once:

  1. A consumer receives a message. The message is not deleted; it becomes invisible to other consumers for a visibility timeout (30 seconds by default in SQS).

  2. If the consumer finishes, it deletes (acks) the message.

  3. If the consumer crashes or is too slow, the timeout expires and the message becomes visible again, so another consumer retries it.

  4. A message that keeps failing (a poison message, such as malformed data) is moved to a dead-letter queue (DLQ) after a maximum number of receives, so it stops blocking work and an engineer can inspect it.

Worked example: retries, duplicates and a DLQ

The simulation below has three messages. Worker W2 processes m2 but crashes before acking, so m2 comes back as a duplicate. Message m3 is malformed and always fails. The consumer keeps a set of processed message IDs, which makes it idempotent.

from collections import deque

VISIBILITY_TIMEOUT = 30          # seconds a received message stays hidden
MAX_RECEIVES = 3                 # after this many failed tries -> dead-letter queue

class Queue:
    def __init__(self):
        self.ready = deque()     # messages waiting to be received
        self.in_flight = {}      # msg_id -> (message, invisible_until)
        self.receives = {}       # msg_id -> how many times it was handed out
        self.dlq = []

    def send(self, msg_id, body):
        self.ready.append((msg_id, body))

    def receive(self, now):
        # messages whose visibility timeout expired become visible again
        for mid, (msg, until) in list(self.in_flight.items()):
            if now >= until:
                del self.in_flight[mid]
                self.ready.appendleft(msg)
        while self.ready:
            msg = self.ready.popleft()
            self.receives[msg[0]] = self.receives.get(msg[0], 0) + 1
            if self.receives[msg[0]] > MAX_RECEIVES:
                self.dlq.append(msg)
                continue
            self.in_flight[msg[0]] = (msg, now + VISIBILITY_TIMEOUT)
            return msg
        return None

    def ack(self, msg_id):       # "delete" after successful processing
        self.in_flight.pop(msg_id, None)

processed = set()                # consumer-side dedupe store (idempotency)
def handle(worker, msg, crash=False, poison=False):
    mid, body = msg
    if poison:
        return f"{worker} FAILED on {mid} ({body})"
    if mid in processed:
        q.ack(mid)
        return f"{worker} skipped duplicate {mid}"
    processed.add(mid)           # side effect: e.g. send the email
    if crash:
        return f"{worker} did {mid} but crashed before ack"
    q.ack(mid)
    return f"{worker} processed {mid} ({body})"

q = Queue()
q.send("m1", "email order #101")
q.send("m2", "email order #102")
q.send("m3", "bad payload")

t = 0
print(t, handle("W1", q.receive(t)))
print(t, handle("W2", q.receive(t), crash=True))     # m2 done, ack lost
print(t, handle("W1", q.receive(t), poison=True))    # m3 fails
for t in (31, 31, 62, 93):                           # time passes, retries
    msg = q.receive(t)
    if msg is None:
        print(t, "queue empty"); continue
    print(t, handle("W3", msg, poison=(msg[0] == "m3")))
print("dead-letter queue:", q.dlq)

Output:

0 W1 processed m1 (email order #101)
0 W2 did m2 but crashed before ack
0 W1 FAILED on m3 (bad payload)
31 W3 FAILED on m3 (bad payload)
31 W3 skipped duplicate m2
62 W3 FAILED on m3 (bad payload)
93 queue empty
dead-letter queue: [('m3', 'bad payload')]

Trace it:

  • At t = 0, m1 succeeds and is acked. m2 is processed but not acked. m3 fails.

  • At t = 31, both unacked messages have passed their 30-second timeout and reappear. m2 is delivered again (at-least-once in action), but the dedupe set recognises it, so the email is not sent twice.

  • m3 fails on its 1st, 2nd and 3rd receive. On the 4th attempt it exceeds MAX_RECEIVES and goes to the DLQ.

In production, the processed-ID set lives in a database (often in the same transaction as the side effect, or as a unique constraint), not in memory.


Ordering

Does the consumer see messages in the order they were sent? Usually only partly, and for good reason: a single global order means a single sequence that every message must pass through, which limits throughput to what one machine can do.

The practical answer is ordering per key:

  • In Kafka, a topic is split into partitions. Order is guaranteed only within a partition. Messages with the same key (for example orderId or userId) go to the same partition, so all events for one order stay in order, while different orders are processed in parallel.

  • In SQS FIFO queues, order is guaranteed per message group ID. SQS standard queues give only best-effort ordering.

  • With competing consumers on a normal queue, even if the queue hands messages out in order, two workers may finish them out of order.

Choose the key so that the things that must be ordered share it ("placed → paid → shipped" for one order), and nothing else is forced into the same line.


Two Kinds of Broker: Queue vs Log

Aspect

Traditional message queue

Distributed log (event streaming)

Examples

RabbitMQ, Amazon SQS, ActiveMQ

Apache Kafka, Amazon Kinesis, Apache Pulsar, Redpanda

After a message is acked

It is deleted

It stays in the log until its retention period expires (Kafka's default is 7 days)

Consumer position

Broker tracks each message's state

Each consumer group tracks an offset per partition

Replay old messages

No

Yes: reset the offset and read again

Routing

Rich (RabbitMQ exchanges route by key patterns, headers)

Simple: topic + partition

Sweet spot

Task queues, per-message retries, complex routing

High-volume event streams, multiple independent readers, event sourcing, analytics pipelines

Worked example: a Kafka-style topic with consumer groups

A topic with 2 partitions. Two consumer groups subscribe: billing (2 instances) and analytics (1 instance). Each group gets every event, partitions are divided among a group's members, and a group can rewind its offsets to replay. (Real Kafka hashes keys with murmur2; we use CRC32 for simplicity.)

import zlib

class Topic:
    """A Kafka-style topic: an append-only log split into partitions."""
    def __init__(self, partitions):
        self.parts = [[] for _ in range(partitions)]

    def publish(self, key, event):
        p = zlib.crc32(key.encode()) % len(self.parts)   # same key -> same partition
        self.parts[p].append((key, event))
        return p

class ConsumerGroup:
    """Every group gets every event; inside a group, partitions are shared out."""
    def __init__(self, name, topic, members):
        self.name, self.topic = name, topic
        self.offsets = [0] * len(topic.parts)            # next position to read, per partition
        self.owner = {p: members[p % len(members)] for p in range(len(topic.parts))}

    def poll(self):
        for p, log in enumerate(self.topic.parts):
            while self.offsets[p] < len(log):
                key, event = log[self.offsets[p]]
                print(f"  [{self.name}] {self.owner[p]} <- p{p}@{self.offsets[p]} {key}: {event}")
                self.offsets[p] += 1                     # commit offset after processing

orders = Topic(partitions=2)
for key, event in [("user-7", "placed"), ("user-3", "placed"),
                   ("user-7", "paid"), ("user-3", "cancelled"), ("user-7", "shipped")]:
    print(f"publish {key}: {event} -> partition {orders.publish(key, event)}")

billing   = ConsumerGroup("billing",   orders, ["bill-A", "bill-B"])
analytics = ConsumerGroup("analytics", orders, ["stats-1"])
billing.poll()
analytics.poll()

print("replay: analytics resets offsets to 0 and reads history again")
analytics.offsets = [0, 0]
analytics.poll()

Output:

publish user-7: placed -> partition 1
publish user-3: placed -> partition 0
publish user-7: paid -> partition 1
publish user-3: cancelled -> partition 0
publish user-7: shipped -> partition 1
  [billing] bill-A <- p0@0 user-3: placed
  [billing] bill-A <- p0@1 user-3: cancelled
  [billing] bill-B <- p1@0 user-7: placed
  [billing] bill-B <- p1@1 user-7: paid
  [billing] bill-B <- p1@2 user-7: shipped
  [analytics] stats-1 <- p0@0 user-3: placed
  [analytics] stats-1 <- p0@1 user-3: cancelled
  [analytics] stats-1 <- p1@0 user-7: placed
  [analytics] stats-1 <- p1@1 user-7: paid
  [analytics] stats-1 <- p1@2 user-7: shipped
replay: analytics resets offsets to 0 and reads history again
  [analytics] stats-1 <- p0@0 user-3: placed
  [analytics] stats-1 <- p0@1 user-3: cancelled
  [analytics] stats-1 <- p1@0 user-7: placed
  [analytics] stats-1 <- p1@1 user-7: paid
  [analytics] stats-1 <- p1@2 user-7: shipped

Notice:

  • All user-7 events land in partition 1, so they are read in order: placed → paid → shipped.

  • Inside billing, bill-A owns partition 0 and bill-B owns partition 1. In a classic Kafka consumer group, each partition is read by at most one member, so the number of partitions is the maximum useful parallelism. A third billing instance would sit idle.

  • analytics independently reads the same events, and later replays them, which a delete-on-ack queue cannot do.


Broker

Model

Key facts

Apache Kafka

Partitioned, replicated log

Very high throughput; per-partition ordering; replay. Since Kafka 4.0 (2025) it runs only in KRaft mode, with no ZooKeeper. Producers are idempotent by default (enable.idempotence=true, acks=all). Transactions give exactly-once for read-process-write pipelines inside Kafka. Share groups ("Queues for Kafka", KIP-932), declared production-ready in Kafka 4.2, add queue-style consumption where many consumers share a partition with per-message acks.

RabbitMQ

Message queue with exchanges

Producers publish to an exchange (direct, fanout, topic or headers), which routes to queues. Replicated quorum queues (Raft) are the choice for durability; RabbitMQ 4.0 removed the old classic mirrored queues. Also offers append-only streams.

Amazon SQS

Fully managed queue

Standard: at-least-once, best-effort order, nearly unlimited throughput. FIFO: ordered per message group, deduplicates retried sends within a 5-minute window. Max message size 1 MiB; retention 4 days by default, up to 14 days; visibility timeout up to 12 hours.

Amazon SNS / Google Cloud Pub/Sub

Managed pub/sub

Push events to many subscribers (queues, HTTP endpoints, functions).

Redis Streams

Log inside Redis

Lightweight streams with consumer groups; good when Redis is already in the stack and volumes are moderate.


Back-of-the-Envelope: Sizing an Event Stream

Using the method from the Back-of-the-Envelope Estimation chapter: 2 billion events a day, 1 KB each, peak = 3× average, 7-day retention, replication factor 3, and one consumer instance can process 5,000 events/s.

import math
events_per_day = 2_000_000_000          # 2 billion order/click events a day
avg_size_bytes = 1_000                  # 1 KB per event
peak_factor = 3
retention_days = 7
replication = 3
per_consumer_rate = 5_000               # events/s one consumer instance can process

avg_rate = events_per_day / 86_400
peak_rate = avg_rate * peak_factor
ingress_mb_s = peak_rate * avg_size_bytes / 1e6
storage_tb = events_per_day * avg_size_bytes * retention_days * replication / 1e12
partitions = math.ceil(peak_rate / per_consumer_rate)

print(f"average rate   : {avg_rate:,.0f} events/s")
print(f"peak rate      : {peak_rate:,.0f} events/s")
print(f"peak ingress   : {ingress_mb_s:,.1f} MB/s")
print(f"disk (7d x 3)  : {storage_tb:,.1f} TB")
print(f"min partitions : {partitions} (one consumer per partition per group)")

Output:

average rate   : 23,148 events/s
peak rate      : 69,444 events/s
peak ingress   : 69.4 MB/s
disk (7d x 3)  : 42.0 TB
min partitions : 14 (one consumer per partition per group)

So we need at least 14 partitions for one consumer group to keep up at peak. Teams usually round up generously (say 24 or 32), because adding partitions later changes which partition a key maps to and can break per-key ordering during the change.


The Dual-Write Problem and the Outbox Pattern

A very common bug: the order service saves the order in its database and then publishes "OrderPlaced" to the broker. If the service crashes between the two steps, the order exists but no email or shipment ever happens. Publishing first has the opposite problem. Two separate systems cannot be updated atomically without help.

Transactional outbox: in the same database transaction that saves the order, insert the event into an outbox table. A separate relay process (or change-data-capture tool such as Debezium) reads new outbox rows and publishes them to the broker, retrying until it succeeds.

BEGIN
  INSERT INTO orders ...                 ┐ one atomic
  INSERT INTO outbox (event) ...         ┘ transaction
COMMIT
        relay: read outbox ──→ publish to broker ──→ mark row sent

The relay may publish a row twice (crash after publish, before marking), so consumers must still be idempotent. At-least-once plus idempotency is the recurring theme.


Watch Out For

  • Consumer lag: if consumers are slower than producers for long, the backlog grows without limit. Monitor queue depth / lag and autoscale consumers on it.

  • Hot keys: if one key (a celebrity's user ID) gets most messages, its partition becomes a bottleneck no matter how many partitions exist.

  • Large payloads: keep messages small. Put the big file in object storage and send its URL (the "claim check" pattern).

  • Eventual consistency: after an async step, other services see the change a little later. Design the UI for it ("Your invoice will be emailed shortly").

  • Do not use a queue when the caller needs the answer now. "Is this password correct?" is a synchronous request, not a message.


Common Mistakes

  • Believing the broker gives exactly-once end to end. Broker features stop at the broker's boundary. Sending an email or charging a card must be made idempotent by your code.

  • Acking before processing and calling it reliable. That is at-most-once.

  • Expecting global order from a partitioned topic or a standard queue.

  • No DLQ: one poison message is retried forever and blocks (or wastes) consumers.

  • Visibility timeout shorter than processing time: the message reappears while the first worker is still busy, causing duplicates.

  • Running more consumers than partitions in a classic Kafka consumer group and expecting more throughput.


Interview Questions

Q1. What is the difference between a message queue and pub/sub?

In a queue (point-to-point), each message is processed by one consumer, and consumers compete to share the work. In pub/sub, each message published to a topic is delivered to every subscriber, each with its own copy. Queues model tasks; pub/sub models events. Systems often combine them: each subscribing service has its own queue (or consumer group) shared by its instances.

Q2. How do you achieve exactly-once processing when your queue is at-least-once?

Make the consumer idempotent. Give every message a unique ID; record processed IDs in a durable store, ideally in the same transaction as the side effect (or rely on a unique constraint or an idempotency key at the downstream API). A redelivered message is then detected and skipped. Inside Kafka, idempotent producers plus transactions give exactly-once for consume-transform-produce pipelines.

Q3. A Kafka topic has 6 partitions and a consumer group has 10 consumers. What happens?

In a classic consumer group, each partition is assigned to at most one consumer in the group, so only 6 consumers receive data and 4 sit idle (as hot standbys). To use more consumers, increase the partition count (keeping in mind that it remaps keys), or, on Kafka 4.2+, consider share groups, which let several consumers share a partition when per-key ordering is not required.

Q4. How do you keep all events of one order in sequence while still processing many orders in parallel?

Use the order ID as the partition key (Kafka) or message group ID (SQS FIFO). Same key → same partition/group → strict order for that order. Different orders spread across partitions and are processed in parallel.

Q5. A service writes to its database and publishes an event. How can these get out of sync, and how do you fix it?

A crash between the two operations leaves one done and the other not (the dual-write problem). Fix it with the transactional outbox: write the event to an outbox table in the same database transaction, then have a relay or CDC publish it. Consumers stay idempotent because the relay may publish twice.


Summary

  • Brokers decouple services, make work asynchronous, absorb spikes, and survive consumer outages.

  • Queue: one consumer per message (competing consumers). Pub/sub: every subscriber gets a copy.

  • Delivery is at-most-once, at-least-once, or exactly-once effect. The usual recipe is at-least-once + idempotent consumers, with a DLQ for poison messages.

  • Ordering is per partition / per key, not global.

  • Queues (RabbitMQ, SQS) delete on ack; logs (Kafka) retain messages, track offsets, and allow replay.

  • Use the outbox pattern to publish events reliably alongside database writes.

Assume every message can arrive late, twice, or out of order, and design consumers so that none of those matters.