08 - Dead Letter Queue and Resilience in Asynchronous Systems

A message that will not go through, and the good ones behind it

In an asynchronous system some messages will fail. The simple case is the transient error: the payment gateway times out, and retrying in a few seconds is enough. The insidious case is the poison pill: a malformed message or one that violates a business rule, for which retrying a thousand times will give a thousand failures, consuming resources and, if the queue is ordered, blocking everything behind it.

The dead letter queue (DLQ) is the answer: after a number of attempts, or at the first unrecoverable error, the message is taken out of the flow and put in a separate queue, where a human or a process can look at it.

A DLQ is not a solution, it is a deferral: it moves the problem from "the system is stopped" to "there are messages to look at", and it works only if someone looks at them. In this article I distinguish the errors you retry from those you do not, I run retries with backoff and a DLQ in a model, I measure what it costs to process batches the wrong way, and I report, from the official documentation consulted on 2026-11-12, the behaviours of SQS, Lambda and EventBridge that come as a surprise.

The context is in the article on SQS, SNS and EventBridge; the reason messages arrive twice is at-least-once delivery, covered in the piece on idempotency. In Kafka the same problem, with retry topics and offset commits, is in the article on the dead letter queue in Kafka.

Retry, wait, give up: the model

The program classifies every error. A TransientError (timeout, service unavailable) deserves another attempt after a wait; any other error (here the SyntaxError of an incomplete JSON) is permanent and sends the message straight to the DLQ, without wasting attempts. The wait grows exponentially: between 0 and base × 2attempt seconds, with a cap, drawn at random (full jitter).

The randomness is there to stop a hundred consumers that failed at the same instant from all retrying together: the AWS Architecture Blog describes adding jitter as the fix that spreads the attempts out. To make the output repeatable the generator has a fixed seed. The effects are protected by an idempotency key: a reprocessed message does not apply its effect twice.

class TransientError extends Error {}

// Repeatable pseudo-random numbers: same seed, same jitter, same output.
let seed = 42;
const random = () => (seed = (seed * 1664525 + 1013904223) % 4294967296) / 4294967296;

// Exponential backoff with "full jitter": random wait between 0 and min(cap, base * 2^attempt).
function wait(attempt: number, base = 1, cap = 60): number {
  return Math.round(random() * Math.min(cap, base * 2 ** attempt) * 10) / 10;
}

type Message = { id: string; body: string };
type DlqEntry = { id: string; body: string; reason: string; attempts: number };
const dlq: DlqEntry[] = [];
const effects = new Map<string, number>();            // how many times the effect was applied, per message
const alreadyDone = new Set<string>();                // idempotency keys

let gatewayDown = 2;        // the gateway fails 2 times and then comes back
let serviceDown = true;     // this service stays down until someone restores it

function handler(m: Message): void {
  const data = JSON.parse(m.body);                    // invalid JSON: SyntaxError, pointless to retry
  if (data.via === 'gateway' && gatewayDown-- > 0) throw new TransientError('gateway timeout');
  if (data.via === 'service' && serviceDown) throw new TransientError('service unavailable');
  if (alreadyDone.has(m.id)) return;                  // idempotency: the effect is applied once
  alreadyDone.add(m.id);
  effects.set(m.id, (effects.get(m.id) ?? 0) + 1);
}

function run(m: Message, maxAttempts = 4): void {
  for (let t = 1; t <= maxAttempts; t++) {
    try { handler(m); console.log(`${m.id}: ok at attempt ${t}`); return; }
    catch (e) {
      if (!(e instanceof TransientError)) {                    // permanent: straight to the DLQ
        dlq.push({ id: m.id, body: m.body, reason: `permanent: ${(e as Error).name}`, attempts: t });
        console.log(`${m.id}: permanent error at attempt ${t}, to the DLQ`); return;
      }
      if (t === maxAttempts) {
        dlq.push({ id: m.id, body: m.body, reason: `transient exhausted: ${(e as Error).message}`, attempts: t });
        console.log(`${m.id}: attempts exhausted (${t}), to the DLQ`); return;
      }
      console.log(`${m.id}: attempt ${t} failed (${(e as Error).message}), waiting ${wait(t)} s`);
    }
  }
}

const queue: Message[] = [
  { id: 'm1', body: '{"via":"local"}' },
  { id: 'm2', body: '{"via":"gateway"}' },
  { id: 'm3', body: '{"via":"local"' },            // the closing brace is missing
  { id: 'm4', body: '{"via":"service"}' },
];
queue.forEach((m) => run(m));
console.log('DLQ:', dlq.map((v) => `${v.id} [${v.reason}, attempts: ${v.attempts}]`).join(' | '));
console.log('alarm DLQ not empty:', dlq.length > 0);

// Remedy: fix the data of m3, restore the service and send the entries back to the main queue.
serviceDown = false;
const toReprocess = dlq.splice(0).map((v) => ({ id: v.id, body: v.id === 'm3' ? '{"via":"local"}' : v.body }));
console.log('--- redrive');
toReprocess.forEach((m) => run(m));
console.log('DLQ after the redrive:', dlq.length, '| effects per message:', Object.fromEntries(effects));

// Output:
// m1: ok at attempt 1
// m2: attempt 1 failed (gateway timeout), waiting 0.5 s
// m2: attempt 2 failed (gateway timeout), waiting 0.4 s
// m2: ok at attempt 3
// m3: permanent error at attempt 1, to the DLQ
// m4: attempt 1 failed (service unavailable), waiting 1.2 s
// m4: attempt 2 failed (service unavailable), waiting 0.9 s
// m4: attempt 3 failed (service unavailable), waiting 3 s
// m4: attempts exhausted (4), to the DLQ
// DLQ: m3 [permanent: SyntaxError, attempts: 1] | m4 [transient exhausted: service unavailable, attempts: 4]
// alarm DLQ not empty: true
// --- redrive
// m3: ok at attempt 1
// m4: ok at attempt 1
// DLQ after the redrive: 0 | effects per message: { m1: 1, m2: 1, m3: 1, m4: 1 }

m2 shows the transient error that resolves: two timeouts and the third attempt goes through. m3 is the poison pill: a single attempt, then the DLQ with the reason and the number of attempts, so as not to lose information. m4 shows the limit of retrying: the service is down for the whole interval, the four attempts run out and the message ends up in the DLQ as "transient exhausted"; it is the same message you would have from a service that will recover, and the DLQ keeps it until that happens.

In the redrive phase the data of m3 is fixed, the service is restored and the entries are sent back to the main queue: both succeed, the DLQ is empty again and each effect is 1, not 2, thanks to the idempotency key. The wait times are written in the log but not waited: it is the virtual time of a model, not a measurement.

The batch: when one bad message takes four good ones down with it

If the consumer receives messages in batches (as a Lambda attached to SQS does), a failure can mean "the whole batch" or "this message". The difference is measured with a batch of five messages, one of them poisoned, and a queue that moves to the DLQ after three receives. In the first variant the function throws an exception at the first error, so no message is deleted; in the second it returns the list of the failed messages only, and the successful ones are deleted.

// Batch of 5 messages, the third is a poison pill. The queue has maxReceive=3: at the third failed receive it goes to the DLQ.
type Msg = { id: string; poison: boolean };
const MAX_RECEIVES = 3;

function simulate(name: string, reply: 'whole-batch' | 'only-the-failed'): void {
  const queue: (Msg & { receives: number })[] = ['a', 'b', 'c', 'd', 'e'].map((id) => ({ id, poison: id === 'c', receives: 0 }));
  const dlq: string[] = [];
  let effectsRun = 0;                            // how many times a "good" message ran its effect
  for (let round = 1; round <= MAX_RECEIVES && queue.length > 0; round++) {
    const batch = [...queue];
    batch.forEach((m) => m.receives++);
    const failed: string[] = [];
    for (const m of batch) {
      if (m.poison) { failed.push(m.id); continue; }
      effectsRun++;                              // without idempotency, the effect repeats on every round
    }
    if (failed.length === 0) { queue.length = 0; break; }
    // With "whole-batch" the function throws an exception: no message is deleted.
    // With "only-the-failed" it returns the list: the successful ones are deleted, the others stay.
    const remain = reply === 'whole-batch' ? batch : batch.filter((m) => failed.includes(m.id));
    queue.splice(0, queue.length, ...remain);
  }
  // After MAX_RECEIVES receives, whatever is left goes to the DLQ.
  for (const m of queue) if (m.receives >= MAX_RECEIVES) dlq.push(m.id);
  console.log(`${name.padEnd(16)} effects of the good messages: ${effectsRun} (expected 4) | in DLQ: [${dlq}]`);
}

simulate('whole batch', 'whole-batch');
simulate('only the failed', 'only-the-failed');

// Output:
// whole batch      effects of the good messages: 12 (expected 4) | in DLQ: [a,b,c,d,e]
// only the failed  effects of the good messages: 4 (expected 4) | in DLQ: [c]

In the "whole batch" variant the four good messages are reprocessed on every round: twelve runs of the effect instead of four, and in the end all five are in the DLQ, the good ones included. The poison pill dragged the healthy messages down with it, and without idempotency it would also have duplicated twelve times what they were supposed to do once.

In the "only the failed" variant the effect runs four times and only c ends up in the DLQ. The Lambda documentation describes it like this: with ReportBatchItemFailures in the mapping configuration and a response with batchItemFailures, only the messages listed become visible again; if the function throws an exception, the whole batch is considered failed. The model is simplified: in the real service the move to the DLQ follows the maxReceiveCount rules described below.

What the documentation says about the real services

These behaviours are not visible in the model and they change the design. They all come from the official AWS guides consulted on 2026-11-12.

• In SQS, maxReceiveCount is the number of times a consumer can receive a message from the source queue before it moves to the DLQ: with a value of 1, a single failure moves it.
• For standard queues the expiry of a message is based on the original enqueue time, which does not change when it moves to the DLQ: with a DLQ retention of 4 days, the message disappears after 3 if it has already spent 1 in the source queue. The guide recommends a DLQ retention longer than that of the source queue.
• For FIFO queues the enqueue time is reset when the message moves to the DLQ, and the guide warns against using a DLQ with a FIFO queue if you do not want to break the exact order of the messages.
• Timers on individual messages (a delivery delay of up to 15 minutes) are not supported by FIFO queues: a backoff based on the delay of the individual message works only on standard queues.
• EventBridge retries by default for 24 hours and up to 185 times; if delivery to a target fails for good, the event is discarded, unless a DLQ is configured on the target.

The second point is the one that does the most damage, because it is silent: a DLQ with a shorter retention than the source queue loses messages exactly when the team needs them most. The fourth takes apart a widespread pattern, retrying with the delay of the individual message on a FIFO queue: it cannot be done; on FIFO the delay is at the queue level. The fifth is the reason why every EventBridge target has a DLQ: without one, after a day of errors the event no longer exists. For the details of EventBridge see the dedicated article.

Watching the DLQ, and reprocessing with method

The most useful alarm signal is the simplest: the DLQ is not empty. The SQS metrics include ApproximateNumberOfMessagesVisible (the messages available to be read) and ApproximateAgeOfOldestMessage, the latter cited by the guide precisely in connection with DLQs. An alarm with a threshold of zero on the first warns you immediately; the one on age tells you whether someone is letting messages rot.

The redrive (sending the messages back from the DLQ to the source queue) is an operation the guide provides for SQS, but the order in which you ran it in the model always holds: first you understand why the message failed, then you fix the data or the code, then you send it back, and the consumer must be idempotent because some of those messages may already have produced a partial effect. Enriching the DLQ entry with the reason and the number of attempts, as the program does, is what makes the first step possible.

Common mistakes

The first is configuring the DLQ and not watching it: messages pile up for weeks and then someone discovers that data is missing. The second is a maxReceiveCount that is too low, which sends healthy messages to the DLQ at every brief unavailability of a service; the third is one that is too high, which delays the discovery of a real fault.

The fourth is mixing permanent errors (corrupted data) and transient ones (service down) in a single DLQ: they have different remedies, and keeping them apart, or at least labelled as in the model, avoids sending back to the queue data that will fail again. The fifth is order: if the flow must stay in order, a DLQ moves it, and you must decide beforehand whether that message can be skipped.

What you can do now

You can separate the errors you retry from those that go straight to the DLQ, you can explain why backoff has jitter and how to reprocess with idempotency. You can measure the cost of a wrong response on the batch (twelve runs and five messages in the DLQ against four and one) and you know the four points of the documentation that you do not see until they hurt: the expiry that starts from the original enqueue, the order on FIFO queues, the timers that FIFO lacks and the 24-hour default of EventBridge. The next step is the foundation that holds all of this up: the idempotency of consumers. The retries and compensations that fail inside a saga are in the article on the saga; what the user sees while messages wait in the queue is in eventual consistency.