Skip to main content
Message Queueslesson 4 of 4 · 3 min read

Kafka vs RabbitMQ Style Brokers

Messages as jobs

RabbitMQ is the classic broker, and in it your messages are jobs.

The broker pushes each one to a consumer, the consumer acknowledges it, and the broker deletes it. Routing is rich, matching messages to queues by keys and patterns, and per-message features like priorities and expiry come as standard. Picture a smart dispatcher handing tasks to workers, which is exactly the right shape for background jobs.

A log wearing a queue costume

Kafka is a different data structure wearing a queue costume. A topic is a log you only ever append to, split into partitions. Messages are written at the end and never deleted when somebody reads them, only aged out by a retention policy.

Your consumers barely interact with the broker at all. Each group simply remembers its own position in the log and moves it forward. Two large things follow from that.

First, you can replay. Because reading destroys nothing, a brand new service can read the topic from the beginning, and a team that ships a bug can wind their position back to before the bad deploy and process it all again. A message acknowledged in the classic model is simply gone.

Second, you get throughput. Appending in order and reading in batches lets Kafka push millions of messages a second, and adding partitions adds parallelism almost in a straight line.

Pay for that with coarser features, ordering only inside a partition, no routing per message, and heavier operations, though a managed offering blunts most of the last one.

Decide with one rule. Take a classic broker for distributing work, jobs that should be done once and forgotten, especially with complicated routing or modest volume. Take Kafka for streams of events, where the messages are facts about what happened, several consumers care, the history has value, and the volume is high.

Expect to run both eventually, and that is usually the correct number of brokers.

the shape of it
Trip serviceKafka topiclog, 7-day retentionPricing groupreads at own paceFraud grouprewound 48hAnalytics groupnew, reads from 0append onceown offsetreplay historyown offset
Kafka consumers pull from the log at independent offsets, so adding or rewinding a consumer costs the producer nothing.

Worked example

A ride-hailing startup begins with RabbitMQ for everything, and it fits: receipt emails, driver payout jobs, document verification tasks. Then the data team asks for every trip event to build surge pricing models, and the fraud team wants the same events, and so does a new analytics vendor. With RabbitMQ, each new consumer means new bindings and another copy of every message fanned out at publish time. Marco, the platform lead, moves trip events to a Kafka topic with 32 partitions and 7-day retention. Producers write each event once; the pricing, fraud, and analytics teams each attach as separate consumer groups reading independently at their own offsets. When fraud ships a broken model in March, they rewind their offset by 48 hours and reprocess 9 million events overnight. Payout jobs never move; they stay on RabbitMQ, where a job done once and deleted is precisely the point.

Message Queues: wrapping up

In the real world

  • 01Kafka was built at LinkedIn to unify its data pipelines and open-sourced in 2011; LinkedIn has publicly described clusters carrying over 7 trillion messages per day.
  • 02AWS SQS, one of the first AWS services ever launched, popularized the visibility timeout model of at-least-once delivery and offers redrive policies that move messages to a DLQ after a configured receive count.
  • 03Shopify has described running Kafka at tens of millions of messages per second during Black Friday Cyber Monday peaks, buffering order and event traffic between services.
  • 04RabbitMQ implements the AMQP protocol and remains the standard backing for job frameworks like Celery, where tasks are acked and deleted rather than retained for replay.
  • 05Stripe requires idempotency keys on its API precisely because payment requests ride at-least-once infrastructure, turning potential double charges from retries into safe no-ops.

Questions people ask

When should I use a message queue instead of a direct API call?

Use a queue when the caller does not need the result to finish its own work: sending emails, generating files, syncing data, notifying other systems. Keep direct calls for anything the user is actively waiting on, like an authorization check during checkout. A useful test is asking whether the operation could happen five minutes late without anyone noticing; if yes, it belongs on a queue.

Why do people say exactly-once delivery is impossible?

Because a consumer can always crash after performing its side effect but before acknowledging the message, and the broker cannot tell that case apart from a crash before the side effect. It must choose between redelivering (possible duplicate) or not (possible loss). The practical answer is at-least-once delivery plus idempotent consumers, which yields exactly-once effects even though delivery itself is not exactly-once.

What metrics should I monitor on a queue?

Queue depth, the age of the oldest message, consumer error rate, and dead letter queue depth. Depth alone can mislead since a fast consumer can handle a deep queue, but a rising oldest-message age means you are falling behind in a way users will eventually feel. DLQ depth should alert at low thresholds, because every message there is promised work that did not happen.

Quick review

Producer → Queue → Consumer. Producer doesn't wait for consumer to process. Decoupled lifecycle
At-most-once:
fire and forget. Message may be lost. Use for metrics, telemetry where loss is acceptable
At-least-once:
delivered ≥1 time. Duplicates possible. Consumer must be idempotent (handle re-delivery)
Exactly-once:
hardest. Requires 2-phase commit or idempotent writes + deduplication. Use for payments
Kafka:
append-only log per partition. Consumers maintain their own offset. Messages retained after consumption (replay). Millions/sec throughput
RabbitMQ:
traditional broker. ACK-based deletion. Routing via exchanges and binding keys. Lower throughput than Kafka
Dead Letter Queue (DLQ):
messages that fail after N retries land here. This is where you debug failures and re-process them
Backpressure:
if consumers are slow, queue grows. Monitor queue depth as key operational metric
the trade-off

Eventual processing. Don't use where you need synchronous response. Adds ops overhead (monitoring, DLQ management).

in the room

Async tasks (email, image processing). Decoupling microservices. Absorbing traffic spikes. Retry infrastructure.

the short version

A log you read at your own pace versus a broker that hands out jobs and deletes them.