Four stages
Four stages connected by queues make up this pipeline, and the queues are the design.
Your producers, the order service, the marketing scheduler, the payment system, publish events. A notification service consumes them, does the thinking, and republishes concrete send tasks onto a topic per channel.
Channel workers, one pool each, consume those tasks and call the outside gateways. Delivery receipts flow back into a status topic that updates your notification log.
Every arrow is asynchronous, so one gateway having a bad afternoon slows its own topic while the others drain normally.
Give the notification service all the thinking. Load somebody's preferences and quiet hours, pick the channels, resolve the template with its variables and language, check the rate limits, and stamp the message with its key.
Keep your channel workers dumb as a result, and dumb is what you want at the edge. Fetch task, call gateway, record result, acknowledge.
Device tokens rot
Give device tokens their own registry service. Tokens arrive when apps register, and go stale when people reinstall or switch phones.
Feed the failures back, because both phone platforms report invalid tokens in their responses and your workers must return those so the registry prunes them.
Skip that feedback loop and you get the classic slow rot. A year in, a third of your pushes go to dead phones and your delivery rate has quietly halved.
Scale per stage, which is boring, and boring is the point. Partition by user so one person's notifications stay in order. Scale each worker pool on its own queue lag, since email tolerates minutes of backlog and a login code does not.
Treat the gateways as your real ceiling. The push platforms want long-lived connections with high concurrency, email providers enforce a per-second quota, and messaging providers meter by account, so tune your concurrency to the contract you actually hold with each one.
Worked example
At a ticketing marketplace, Elena's pipeline handles a Taylor Swift onsale. At 10:00 the waiting-room service publishes 4 million your queue position events over 20 minutes, while normal traffic (receipts, delivery reminders) continues underneath. Kafka absorbs the burst; consumer lag on the push topic peaks at 90 seconds, which product accepted in advance for queue-position updates. The instructive failure comes at 10:07: FCM starts returning 502s for 3 minutes. Because channel workers are isolated, email receipts keep flowing; the push workers back off, lag grows to 6 minutes, then drains at 40,000 sends per second once FCM recovers. Her favorite graph in the retro shows the SMS topic, reserved for payment confirmations, sitting at 200 ms of lag through the whole event because its worker pool and quota were never shared with the flood.