Exactly once does not exist
Start from an honest premise. Delivering exactly once to an outside gateway does not exist.
Picture the ambiguity. Your worker calls the provider and the connection drops before the answer arrives, so the message may or may not have gone, and no protocol can tell you which.
Take the achievable contract instead: at least once, with deduplication. Retries become mandatory, and duplicates become harmless instead of impossible.
Retries, and where they stop
Retry in the standard shape, backing off with randomness, one second then two then four then eight, capped at a handful of attempts.
Classify your errors before retrying, because retrying a client error is a bug. An invalid token or a malformed address fails identically forever, and only rate limits, server errors and timeouts deserve another go.
Honour any wait header the provider sends, and wrap each gateway in a circuit breaker. When a provider is down hard, your workers fail fast and the queue holds the backlog instead of burning attempts into a dead endpoint.
Send whatever exhausts its retries to a dead letter queue, and then actually operate that queue. Alert on its depth, keep the failure reason and the full payload with every entry, and build the replay tool before the incident, not during it.
Set the replay policy per priority. When the outage ends, somebody replays and deliveries complete hours late, which for a newsletter is a shrug and for a password reset code is worthless. So let the transactional codes expire and replay the rest.
Run idempotency, meaning the property that handling the same message twice changes nothing, on two layers. The caller's key stops duplicate events at the front door, checked against a store with an expiry, so an order service that crashes and fires the same event three times produces one notification.
Check again inside the pipeline, where your worker looks for a sent-marker for this notification and channel before calling out.
Accept the residual race, which is why the contract says at least once. The marker written just before a crash, or the ambiguous timeout above, still allows the rare duplicate. One duplicate message a week is an annoyance. A dropped one is a support ticket.
Worked example
A neobank's notification system meets its worst night when SendGrid has a 40-minute outage at 2 am. The pipeline behaves: workers retry with backoff, the circuit opens after 90 seconds of consecutive failures, and 380,000 emails accumulate across the retry topic and DLQ. The bug is upstream. The statements service, seeing no delivery confirmations, re-emits its events at 2:30, and it does not send idempotency keys, so when SendGrid recovers, 140,000 customers get their monthly statement email twice. Complaints are mild, but the duplicate-send graph gets shown in every design review afterward. Fixes: idempotency keys become mandatory (requests without them are rejected), the API's deduplication store keeps keys for 48 hours in Redis, an in-memory store, and DLQ replay gets a dry-run mode showing what would be sent, per channel, before anyone presses the button.