JA

Understanding the difference between Pub/Sub and queues

Introduction

Lately my interest has shifted away from the “technology that makes things run” — frameworks, middleware — and toward how to choose an architecture that fits the complexity of the domain. While a domain is simple, you can just write one application in a straightforward way. As the business and the number of parties involved grow, how services connect to each other moves to the center of the design.

The typical way to connect them is asynchronous messaging, which is also the foundation you build an event-driven architecture on. The two names that come up most are the queue (message queue) and Pub/Sub (publish/subscribe). Both decouple producers from consumers, but they serve different purposes in terms of who a message is delivered to and how many receive it.

In short, a queue is for splitting work across multiple workers; Pub/Sub is for telling multiple parties that something happened. The Azure Service Bus documentation frames it the same way: queues are point-to-point, while topics and subscriptions are one-to-many publish/subscribe.

Here is what this post covers:

What a queue is

A queue temporarily holds messages that a producer puts in, and consumers take them out and process them. Even when several consumers watch the same queue, each message is normally processed by only one of them.

Producer
   │
   ▼
 Queue ──┬──▶ Worker A
         ├──▶ Worker B
         └──▶ Worker C

Each delivery is processed by exactly one of the workers

This is also known as the Competing Consumers pattern. Adding workers does not make them run the same job twice; it lets them share the backlog that has piled up in the queue.

Typical fits for a queue:

  • Image and video conversion
  • Sending email
  • PDF generation
  • Batch processing
  • Calls to external APIs
  • A buffer to absorb sudden spikes in traffic

AWS’s own comparison likewise describes Amazon SQS as suited to work queues, batch processing, and buffering events.

What Pub/Sub is

In Pub/Sub, the publisher does not send messages directly to consumers; it publishes them to a topic. Each subscriber receives messages through a subscription attached to that topic.

                 ┌─▶ Subscription A ─▶ Inventory service
Publisher ─▶ Topic
                 ├─▶ Subscription B ─▶ Email service
                 └─▶ Subscription C ─▶ Analytics service

The same event is delivered to every subscription

When a topic has several subscriptions, each published message is copied to every one of them. The publisher does not need to know how many subscribers there are or who they are, so senders and receivers stay decoupled (Google Cloud Pub/Sub documentation).

For example, when the order service publishes an OrderCreated event, the inventory service can decrement stock, the email service can send an order confirmation, and the analytics service can record the sale. When a new kind of processing is needed, you add a new subscription and consumer — the publisher does not change.

The differences, side by side

AspectQueuePub/Sub
Communication modelPoint-to-pointPublish/subscribe
Main relationshipOne-to-oneOne-to-many
Who receives a messageUsually one of many consumersEvery subscription gets its own copy
Why you add consumersLoad distribution, parallelismNew processing, new parties to notify
Main useRunning jobs and tasksEvent notification, fan-out
ExamplesImage conversion, email, batch jobsOrder created, user signed up, product updated

The key point is that queues pair well with commands and tasks, while Pub/Sub pairs well with events. If a message says “convert this image” — something you want done — a queue is the natural fit. If it says “an order was created” — a fact that already happened and several services should hear about — Pub/Sub is the natural fit.

Combining the two

Pub/Sub and queues are not mutually exclusive; real systems combine them. From the receiving side, each subscription on a topic behaves like a virtual queue: the same event is shared across subscriptions, while within one subscription several consumers split the work.

Topic
  ├─▶ Subscription A
  │      ├─▶ Worker A1
  │      └─▶ Worker A2
  │
  └─▶ Subscription B
         ├─▶ Worker B1
         └─▶ Worker B2

Here, every event reaches both subscription A and subscription B. Within subscription A, though, each message is handled by either A1 or A2. In other words, you get Pub/Sub fan-out across subscriptions and queue-style load distribution within a subscription. Google Cloud Pub/Sub works this way too: a topic delivers each message to every subscription, and when one subscription has several subscribers, only one of them receives any given message.

An example on AWS

On AWS, the classic setup combines Amazon SNS as the Pub/Sub topic with Amazon SQS as the queues.

SNS Topic
  ├─▶ SQS Queue A ─▶ Inventory service workers
  ├─▶ SQS Queue B ─▶ Email service workers
  └─▶ SQS Queue C ─▶ Analytics service workers

A message published to the SNS topic fans out to every SQS queue subscribed to it. Each service processes messages from its own queue at its own pace, which isolates failures and differences in throughput between services. The Amazon SNS FAQ also explains that subscribing multiple SQS queues to an SNS topic delivers the same message to all of them.

Design pitfalls

Introducing a queue or Pub/Sub does not guarantee that each message is processed exactly once. Depending on the service and its configuration, the same message may be redelivered, so consumers need to be idempotent — processing a message more than once must not corrupt the result. Google Cloud Pub/Sub, for instance, uses at-least-once delivery by default and redelivers any message that was not acknowledged.

Retrying failed messages forever can also crowd out healthy work. So you design retry limits, exponential backoff, a dead-letter queue, and monitoring with alerts. If message order matters, do not assume it is implicitly guaranteed across a whole queue or topic; check the ordering guarantees of the service you use and the unit (partition) they apply to.

How to choose

These questions make the decision easier:

  • “I just need someone to handle this job once” → a queue
  • “I want several services to know this happened” → Pub/Sub
  • “I want several services to know, and have each one split the work across its workers” → Pub/Sub combined with queues

If you think of queues as sharing out work and Pub/Sub as sharing information, the two roles are much harder to mix up. That said, in some products a subscription itself behaves like a queue, so it helps to keep the conceptual patterns separate from the cloud services’ product names.

References