Skip to main content
Pub/Sub Patternlesson 2 of 3 · 3 min read

Fan-Out Mechanics

A queue per subscriber

Fan-out sounds like your broker simply sends several copies, and the durable version has a more specific shape worth knowing.

See the problem with plain broadcasting. Your subscribers fail independently. If the fraud service is down for an hour, its messages have to wait somewhere, without touching the email service and without the publisher caring at all.

Give each subscriber its own queue, the shape the cloud providers sell. Publish to a topic, and the topic drops each message into a separate queue per subscriber. Each of those is that subscriber's private durable buffer, with its own retry policy and its own dead letter queue.

Fraud being down now just means fraud's queue gets deeper. Nobody else notices. Kafka reaches the same place differently: the log itself is the buffer, and each consumer group's position is its private bookmark.

Contrast that with a fire-and-forget broadcast, where a message goes only to whoever is connected at that instant and anyone disconnected misses it permanently. Fine for a throwaway signal like clearing a cache. Disqualifying for anything you would call a business event.

The famous version: feeds

Meet the famous version of this problem at the application level: social feeds. Push each new post into every follower's feed as it is written and reads are instant, until a celebrity with 50 million followers turns one post into 50 million writes.

Store the post once and build feeds when they are requested and you flip the cost: cheap writes, expensive reads.

Take the hybrid that large social products documented years ago. Normal accounts push on write, celebrity posts get merged in on read.

Generalise the lesson, because it outlives feeds. Fan-out cost scales with how many subscribers you have, and past some multiple you change strategy rather than buying more of the same.

the shape of it
Listing svcSNS topiclisting.updatedSearch queuePricing queuebuffers downtimeSearch indexerPricing svcdown, recoverspublish oncecopycopyconsumedrain on return
step 1 of 3
Topic-to-queue fan-out gives every subscriber a private durable buffer, so one team's outage stays theirs.

Worked example

Ana's team at a marketplace publishes listing.updated events straight from SNS to each subscriber's HTTPS endpoint. During a Black Friday load test, the search indexer starts timing out, SNS retries hammer it harder, and by the time it recovers, thousands of deliveries have exhausted retries and vanished; search shows stale prices for two days. The rebuild follows the standard pattern: each of the five subscribers gets its own SQS queue subscribed to the topic, with a 4-retry redrive policy into a per-team DLQ. On the next load test the indexer falls behind by 80,000 messages during the peak, then drains its queue in 12 minutes once traffic drops, losing nothing. The pricing team, down for a deploy during the same test, comes back to an intact backlog. Ana's summary in the retro: the topic broadcasts, but every subscriber needs its own queue to fail privately.