Skip to main content
Notification Systemlesson 4 of 4 · 3 min read

Preferences, Rate Limits, and Priority

Preferences have legal teeth

Treat preference checking as correctness with legal teeth. Sending marketing email to somebody who opted out breaks the law, not merely taste.

Model it as a matrix of category against channel. Transactional, security, social and marketing, crossed with push, email and messages, with security notifications not optional.

Put the check in your notification service, on every single send, reading from a cached store.

Never let callers assert that this person opted in, because that is exactly how compliance incidents happen. Callers cache stale answers.

Understand who your rate limits protect, because it is your users, from you. Product teams ship notifications independently, and the sum is fourteen pings a day until people switch notifications off entirely, which surveys repeatedly name as a leading reason for deleting an app.

Cap per person per category per day. Three social pushes a day, marketing weekly. Mechanically that is one counter keyed by person, category and day, incremented and compared per send, which is trivial to build.

Expect the organisational fight over whose notification gets cut to be the hard part. That is why mature systems evolve toward one central scheduler that batches, times and drops notifications per person across every producer, and at least one large company has published theirs.

Hang quiet hours and batching off the same decision point. Hold anything non-urgent during somebody's local night, and collapse five people liked your post into one digest.

Give both a scheduler with a store for delayed delivery, rather than pure pass-through queues.

Priority lanes

Divide your lanes by priority last. A password reset code and a sale announcement must never share a queue, because a campaign burst delays that code by eleven minutes exactly when somebody is staring at a login screen.

Run separate topics per tier with their own worker pools. Provision the transactional lanes with headroom for bursts, and throttle marketing to a drain rate that cannot starve anybody.

Tie those lanes back to your earlier burst arithmetic in an interview. That is what makes the whole design cohere rather than sounding like a list of features.

the shape of it
EventsNotification serviceprefs, caps, quiet hoursTransactional laneheadroom keptMarketing lanethrottled1. all producers2. reset codes3. campaigns
step 1 of 3
Separate lanes stop a 20 million campaign delaying somebody's password reset.

Worked example

An e-commerce app's uninstall rate climbs for two quarters, and exit surveys keep saying too many notifications. Ines audits a week of sends for a sample user: 26 notifications, including 4 for one order (confirmed, packed, shipped, delivered), 9 marketing pushes from three teams unaware of each other, and price alerts at 3 am for a user in Mumbai because the scheduler ran in UTC. She ships three changes: a per-user cap of 2 marketing pushes daily enforced with Redis counters keyed (user, category, day), quiet hours computed from the device timezone with holds released at local 9 am, and collapsing of order-status pushes so packed silently replaces confirmed if it is still unread. Marketing predicts a conversion drop; over the next quarter, notification opt-out rate falls from 31 to 22 percent and click-through per marketing push nearly doubles, because the pushes that survive are the ones users tolerate.

Notification System: wrapping up

In the real world

  • 01LinkedIn built Air Traffic Controller, a centralized service its engineering blog describes as deciding channel, timing, and frequency per member across all notification producers, after uncoordinated teams collectively over-notified users.
  • 02Apple's APNs invalidates device tokens on app uninstall and reports them in responses, and requires providers to stop sending to them, which is why serious systems maintain a token registry with a pruning feedback loop.
  • 03Uber has described its notification platform handling hundreds of millions of messages a day across push, SMS, and email, with per-channel pipelines and provider failover, for example routing SMS through alternate aggregators by country when one degrades.
  • 04Slack batches and debounces notifications deliberately: it waits to see if you read a message on desktop before pinging your phone, a published example of notification decisioning that goes far beyond fire-and-forget.
  • 05FCM offers collapse keys so a newer notification replaces an older undelivered one on the device, platform-level support for the same duplicate-suppression thinking the pipeline needs internally.

Questions people ask

Why not guarantee exactly-once delivery?

Because the last hop makes it impossible: when a call to APNs or Twilio times out, you cannot know whether the message went out, so any retry risks a duplicate and any non-retry risks a drop. Systems therefore promise at-least-once and invest in dedup, idempotency keys at the API, sent-markers before gateway calls, and collapse keys on the device, so the rare duplicate is harmless.

How should priorities be implemented, one queue with priority fields or separate queues?

Separate queues (or Kafka topics) per priority tier, each with its own worker pool. A priority field in a single queue does not help once a 20-million-message campaign is physically ahead of your password reset in the partition. Separate lanes give transactional traffic dedicated drain capacity that marketing bursts cannot consume.

Where do user preferences get enforced?

In the notification service, on every send, against the live preference store. Producer teams should not even be able to express send this regardless of preferences, except for a narrow security category. Centralizing the check is what makes compliance auditable: one code path, one log, one place to prove an opted-out user was never contacted.

Quick review

Producers (order service, social service) emit events to Kafka topics
Notification service consumes events → queries user preferences DB → routes to correct channel workers
Channel workers call third-party:
APNs (iOS push), FCM (Android push), SendGrid/SES (email), Twilio (SMS)
Retry with exponential backoff (1s, 2s, 4s, 8s...). Max 3 retries. Dead Letter Queue for permanently failed
Rate limiting per user per channel:
don't send 100 emails/day to same user. Redis counter per (user, channel, day)
User preferences:
stored in DB. User can opt-out per category. Check before every notification
Idempotency:
track notification_id to prevent duplicate delivery on retry. Check before calling third-party
Priority queues:
critical alerts (password reset, payment failed) in high-priority queue. Promotional in low-priority
the trade-off

At-least-once → possible duplicates. Exactly-once requires distributed transactions. Accept duplicates, make them idempotent.

in the room

Tests async pipeline design, third-party integration, reliability (retry/DLQ), fan-out, and user preference management.