Retries and Dead Letter Queues

Two species of failure

Failures in your consumer come in two species, and the entire retry design falls out of telling them apart.

Transient ones succeed if you simply wait and try again: a database failing over, an API rate limiting you, a network blip. Permanent ones will fail every single time no matter how patient you are: a malformed message, a bug that throws on one input, a user who no longer exists.

Retry the transient ones, backing off as you go, so a struggling dependency is not set upon by an angry mob of retrying consumers. Wait a second, then two, then four, then eight, with a little randomness so your retries do not all fire together.

Watch what those same retries do to a permanent failure. The message gets redelivered forever, and depending on your broker it can sit at the head of the queue failing in a loop while healthy messages pile up behind it. That is a poison pill, and one bad message stalling a million good ones is a genuinely classic outage.

The escape hatch

Give it an escape hatch. After a handful of failed attempts, your broker moves the message onto a separate queue instead of redelivering it, and your main queue keeps flowing.

Use that dead letter queue as a workbench. Inspect the message that failed, fix the bug or the data, and replay it through the main queue. Most teams end up building a small replay tool for exactly this.

Follow two operational rules or it will not work. Alert on how deep the dead letter queue is getting, because one filling up quietly is data loss on a timer, and those messages are work your system promised somebody it would do.

And log why each message died, with the attempt count and the last error attached. A dead letter queue full of anonymous failures at 3am is barely better than not having one.

void handle(Message m) {
  try {
    process(m);
    m.ack();
  } catch (TransientException e) {
    // Database failing over, API rate limiting, network blip.
    // Wait longer each time, with randomness so every consumer
    // does not come back in the same instant.
    long backoff = (1L << m.attempts()) * 1000;          // 1s, 2s, 4s, 8s
    long jitter  = ThreadLocalRandom.current().nextInt(1000);
    m.retryAfter(Duration.ofMillis(backoff + jitter));
  } catch (PermanentException e) {
    // Malformed payload, deleted user. This will fail identically
    // forever, so retrying it is a bug, and leaving it at the head
    // of the queue stalls every healthy message behind it.
    deadLetter.send(m, e.getMessage(), m.attempts());
    m.ack();
  }
}

Worked example

Nadia's team ingests product updates from merchants into a catalog service via SQS. One merchant's integration starts sending a price field as a string with a currency symbol, and the consumer throws on parse. With maxReceiveCount not configured, the message redelivers every 30 seconds, and each attempt occupies a worker slot; by morning the queue backlog is 340,000 messages and catalog updates are 6 hours stale. The fix comes in layers: a redrive policy moves any message to a DLQ after 4 attempts, an alarm fires when DLQ depth exceeds 50, and the consumer logs the offending merchant ID with each failure. The bad merchant's 212 messages land in the DLQ, the backlog drains in 40 minutes, and after the parser learns to strip currency symbols, Nadia replays all 212 through the main queue with a 30-line script.