Three layers of reliability
Webhook delivery is a distributed system with all the usual failure modes and none of the usual control, because the sender is another company. You build reliability in three layers.
Lean on the provider's retries first. On anything other than a success, serious providers back off and try again, some for as long as three days.
That covers your deploys and your blips, and it is exactly why acknowledging fast and processing later matters. A handler that does the work inline and answers slowly converts its own latency into a retry storm.
Handle the duplicates those retries create, because a delivery that worked but whose acknowledgement got lost in transit comes again. Store every provider event identifier and skip the ones you have seen, ideally with a uniqueness constraint so your database enforces it and your code does not have to.
Own your durability second. Once you return a 200, the provider considers that event delivered forever.
So a 200 followed by losing the event, a crash before processing, a bug that drops it, cannot be recovered from their side at all. That is why your endpoint writes to a queue or a table before answering: your 200 should mean stored durably, not merely received. Processing then retries from your own queue, with its own dead letter queue for events that keep failing.
The layer teams skip
Add reconciliation third, which is the layer teams skip. Even excellent providers miss deliveries sometimes, when your endpoint is down past their retry window, or something was misconfigured, or an incident on their side ate an event.
Run a job that periodically asks the provider to list events, compares that against what you received, and backfills whatever is missing.
Read the division of labour there. Webhooks are your fast path and polling is your audit. Teams that reconcile find their misses in a report. Teams that do not find them in support tickets.
Worked example
Farah's team at a B2B invoicing product handles Stripe webhooks inline: the handler updates the database, calls the email provider, then returns 200, typically in 8 seconds. During a Tuesday incident their email vendor slows down, the handler starts exceeding Stripe's timeout, and Stripe begins retrying. The retries land on the same slow handler, tripling load, and some invoices are marked paid twice thanks to double processing. Post-incident, the team rebuilds in layers: the endpoint verifies, inserts the event into a webhook_events table with a UNIQUE constraint on the Stripe event ID, and returns 200 in 40 ms; a worker pool processes rows async with retries and a dead-letter state. A nightly job pulls /v1/events for the last 48 hours and reconciles, and in its first month it backfills 6 events that never arrived, each one a paid invoice that would have stayed unpaid in the UI.