Idempotencylesson 3 of 3 · 3 min read

Designing Idempotent Operations

Make the operation safe instead

Keys are bookkeeping bolted onto an unsafe operation. Often you can make the operation itself safe to repeat instead, which means less machinery and fewer ways to fail.

Reach first for stating destinations instead of movements. Add 50 to the balance, applied twice, is corruption. Set this order to shipped, applied twice, is a Tuesday.

Whenever your operation can name the end state rather than the change, repeats collapse into doing nothing for free. Insert-or-update keyed on a natural identifier follows the same logic, so replaying a create-user event simply rewrites the same row.

Guard the changes you genuinely cannot restate as destinations. A conditional write is the standard tool: a version column, or a condition saying apply this only if the state is still what I expect. Your replay finds the condition false and does nothing.

Get the same guard with better ergonomics from a state machine. Allow shipped only from paid, and a redelivered ship-this-order event finds the order already shipped and falls straight through.

Queue consumers

Pay special attention to queue consumers, because at-least-once delivery makes duplicates routine. A consumer crashing after processing and before acknowledging guarantees a redelivery.

Use a table of processed events. Insert the event identifier and apply the state change inside the same database transaction, so a duplicate hits the uniqueness constraint and the transaction aborts cleanly.

The same-transaction detail is the entire trick. Tracking those identifiers in a separate store reopens the exact crash window you were trying to close.

Some side effects resist all of this. You cannot un-send an email or make a messaging provider forget what it delivered.

Push the protection to the boundary there. Record the intent to send under a deduplication key first, and have a single sender consume those records. You end up sending at most once, guarded by recording at least once, which is usually the right trade for notifications.

the shape of it

step 1 of 4

Recording the id and applying the change in one transaction is the whole trick.

Worked example

Rohan's warehouse system consumes stock events from Kafka: received, picked, shipped, each adjusting an inventory count. Physical audits keep finding the numbers drift high by around 3 percent a week, always high, never low. The cause is consumer group rebalances: whenever a pod dies or deploys, a batch of events gets redelivered and the handler happily increments the same counts again. His fix is one migration and ten lines of code: a processed_events table with a unique constraint on event_id, and the insert wrapped in the same transaction as the count update. Redelivered events now die on the constraint before touching inventory. The next monthly audit comes back within 0.1 percent, and the team retires the Sunday recount shift that had quietly existed to paper over the drift.

Idempotency: wrapping up

In the real world

01Stripe accepts an Idempotency-Key header on POST requests, stores the result of the first call for 24 hours, and replays the saved response to any retry with the same key, so a network blip cannot double-charge a card.
02Kafka's idempotent producer, on by default since Kafka 3.0, attaches a producer ID and sequence number to every batch so brokers can discard duplicates created by producer retries.
03AWS SQS standard queues deliver at-least-once by design and the documentation tells consumers to be idempotent; FIFO queues add broker-side deduplication over a 5-minute window keyed by MessageDeduplicationId.
04PayPal's REST API takes a PayPal-Request-Id header so a retried payment creation returns the original payment instead of creating a second one.
05DynamoDB's TransactWriteItems accepts a ClientRequestToken that makes an entire multi-item transaction idempotent for 10 minutes, aimed exactly at the ambiguous-timeout retry case.

Questions people ask

If GET, PUT, and DELETE are already idempotent, why do I need idempotency keys?

The HTTP spec describes how those methods are supposed to behave, but the guarantee only exists if your handler implements it, and the operations that hurt most when duplicated, like charging a card or creating an order, are POSTs with no method-level protection. Keys give any operation, regardless of verb, a server-side way to recognize a repeat.

How long should the server remember idempotency keys?

Long enough to cover every realistic retry: client backoff schedules, queue redelivery, and webhook retry windows. Stripe uses 24 hours and that is a good default. Longer retention costs storage and buys protection against ever-rarer late retries; shorter retention risks a late retry being treated as a brand-new operation.

Why not just use a queue with exactly-once delivery instead?

End-to-end exactly-once delivery does not exist in a distributed system, because the acknowledgment itself can be lost, leaving the broker unsure whether you processed the message. Systems that advertise exactly-once semantics, like Kafka transactions, build them from at-least-once delivery plus idempotent processing. That combination is the practical equivalent, and the idempotent half is your job.

Quick review

HTTP:: GET, PUT, DELETE are idempotent by definition. POST is not. Submitting twice creates two resources
Idempotency key:: client generates unique UUID per operation, sends in header (Idempotency-Key). Server stores result keyed by it
On retry:: server checks if idempotency key was seen. Returns cached result instead of re-executing
Storage:: store idempotency key → result in Redis with TTL (24 hours typical). Check atomically with operation
Payment critical:: charging a user twice is catastrophic. Stripe, Braintree, Adyen all require idempotency keys for charges
At-least-once delivery:: message queues may redeliver. Consumers must be idempotent by design
Deduplication:: track processed event IDs in a DB or bloom filter. Reject duplicates without re-processing

the trade-off

Storage and lookup overhead per operation. Worth it for any money-moving or state-mutating operation.

in the room

Any operation that might be retried: network timeouts, message redelivery, user double-clicks, webhook retries.

the short version

The cheapest dedup is an operation that is safe to repeat by construction.