
A single malformed order can stop an entire Kafka partition. The consumer retries the same record, never advances its offset, and every healthy order behind it waits. The result is a growing lag alert caused by one poison message, not by insufficient consumer capacity. A dead letter queue gives the failure a durable place to go without pretending the message was successfully processed.
This guide explains the Kafka dead letter queue pattern: how to classify failures, choose bounded retries, preserve useful metadata, route a poison message safely, and replay it after the underlying problem is fixed. Continue from Kafka consumer lag, Kafka offsets, and Kafka delivery guarantees. The Backend Developer Interview Guide connects DLQs to the broader messaging and reliability sequence.
Table of Contents
Open Table of Contents
- What a Kafka Dead Letter Queue Is
- Classify the Failure Before Retrying
- Retry Topics and Backoff
- Designing the DLQ Record
- Routing and Committing Safely
- Inspecting and Replaying Messages
- Retention, Security and Monitoring
- Production Example
- Interview Questions
- 1. What is a Kafka dead letter queue?
- 2. When should a message go to a DLQ instead of being retried?
- 3. What metadata belongs in a DLQ record?
- 4. How do you avoid losing a message while writing to the DLQ?
- 5. How should DLQ messages be replayed?
- 6. Can a DLQ solve a hot partition or high consumer lag?
- Conclusion
- References
- YouTube Videos
What a Kafka Dead Letter Queue Is
In Kafka, a DLQ is normally a dedicated topic where a consumer or connector writes records that cannot be processed on the normal path. The broker does not generally inspect an exception and move the record for a standard consumer; the application, Kafka Connect, or Kafka Streams error handler decides what to do. This distinction matters because the application owns the retry budget, metadata, permissions, and replay policy.
A DLQ is not a trash can and it is not proof that a record was delivered successfully. It is a durable quarantine with an operational contract:
- Preserve enough of the original record and failure context to diagnose it.
- Stop blocking healthy records when the failure is permanent or the retry budget is exhausted.
- Alert an owner and define who can inspect, repair, discard, or replay it.
- Replay only after the consumer and sink are safe to run again.
flowchart TD
A[Source topic] --> B[Consumer poll]
B --> C{Process record}
C -->|Success| D[Commit source offset]
C -->|Transient failure| E[Bounded retry or retry topic]
E --> C
C -->|Permanent or poison failure| F[Publish DLQ record]
F --> G{DLQ publish acknowledged?}
G -->|Yes| H[Commit source offset]
G -->|No| I[Do not advance; retry safely]
H --> J[Alert and investigate]
Kafka Connect and Kafka Streams have framework-specific error-handling capabilities, while a plain KafkaConsumer requires application code. The pattern is the same in each case: do not acknowledge the source record until the chosen failure destination is durable.
Classify the Failure Before Retrying
Retrying every exception is a reliability bug. First classify the failure by whether another attempt can change the outcome:
| Failure class | Examples | Action |
|---|---|---|
| Transient | Database timeout, connection reset, 429, temporary dependency outage | Retry with bounded backoff and jitter |
| Permanent data error | Invalid schema, missing required field, impossible enum value | Route to DLQ after validation; do not retry forever |
| Poison message | Deserializer crash, malformed bytes, bug triggered by one payload | Quarantine with original bytes and failure metadata |
| Business rejection | Duplicate order, expired command, failed validation rule | Decide whether a rejected-events topic is clearer than a DLQ |
| Infrastructure-wide outage | Broker unavailable, database cluster down | Pause or back off; a DLQ can hide an outage |
The same exception can change class during an incident. A 503 is transient until the retry budget is exhausted; a schema version that the deployed consumer cannot decode is permanent until the consumer is upgraded. Make the classifier observable by recording the failure category, not only the stack trace.
Do not send a record to the DLQ just because processing is slow. A slow but valid record belongs in capacity and timeout analysis, while an invalid record belongs in an explicit quarantine path. Kafka consumer lag explains why blindly retrying a poison message can look like a scaling problem.
Retry Topics and Backoff
In-place retries keep the source partition blocked until the record succeeds or is routed away. That preserves strict partition order, but a long outage can stop unrelated keys sharing the partition. A retry topic lets the main consumer continue, but it requires an ordering decision and a way to delay redelivery.
| Strategy | Strength | Risk |
|---|---|---|
| In-place bounded retry | Simple; preserves source partition order | One failing record blocks records behind it |
| Retry topic with delay buckets | Keeps the source consumer moving | Replayed records can change timing and ordering |
| Retry topic per attempt | Clear retry count and operations | More topics, retention, and routing overhead |
| Immediate DLQ | Fast isolation for permanent errors | Loses automatic recovery for transient errors |
Kafka topics do not natively provide a per-record “deliver after 30 seconds” primitive. Common designs use retry topics such as orders.retry.1m, orders.retry.10m, and orders.retry.1h, or a scheduler that waits until the record timestamp is eligible. Do not busy-loop on a retry topic: that burns consumer capacity and creates a second failure storm.
flowchart TD
A[Source record] --> B{Retryable?}
B -->|No| C[DLQ immediately]
B -->|Yes| D[Attempt 1]
D --> E{Succeeded?}
E -->|Yes| F[Commit or transaction complete]
E -->|No| G[Retry 1m]
G --> H[Attempt 2]
H --> I{Succeeded?}
I -->|Yes| F
I -->|No| J[Retry 10m]
J --> K{Budget exhausted?}
K -->|No| H
K -->|Yes| C
Carry the original event ID and attempt count across retry topics. A retry hop must not create a new business event ID, or the sink cannot distinguish one failing event from several different events. Use a retry budget based on attempts, elapsed time, or both so an outage does not create unbounded retention.
Designing the DLQ Record
A useful DLQ record preserves the original key and value whenever the bytes are safe to retain, then adds failure context in headers or a typed envelope. At minimum, capture:
- Original topic, partition, offset, and Kafka timestamp.
- Consumer group, application name, version, and deployment ID.
- Stable event ID, original key, retry attempt, and first-failure time.
- Failure category, exception class, sanitized message, and stack-trace reference.
- Trace or correlation ID and remediation or replay status.
Keep sensitive payload fields protected. Error messages and stack traces can contain tokens, customer data, or database values. Restrict DLQ readers and writers with ACLs, encrypt the topic as required, and redact logs even when the DLQ stores original bytes.
{
"originalTopic": "orders.v1",
"originalPartition": 7,
"originalOffset": 184203,
"eventId": "evt-8c21",
"attempt": 4,
"failureType": "SCHEMA_VALIDATION",
"exceptionClass": "OrderSchemaException",
"failedAt": "2026-09-19T20:18:37Z",
"consumerGroup": "inventory-v2",
"applicationVersion": "inventory-2026.09.19.2",
"traceId": "trace-42",
"replayStatus": "PENDING"
}
Use the original event key when publishing the DLQ record if operators need to correlate failures by aggregate. Matching the source partition count helps preserve partition-to-partition diagnosis. A shared cluster-wide DLQ may be cheaper, but it mixes schemas, owners, and retention policies; one DLQ per application or consumer group is often easier to govern.
Do not compact a DLQ by default. Compaction can remove earlier failures for the same key, erasing the first record needed to explain a later state. Use delete retention with an explicit duration, or archive records to controlled object storage before expiry.
Routing and Committing Safely
The source offset must advance only after the success path or the chosen failure destination is durable. If a consumer writes a DLQ record and commits before Kafka acknowledges the write, a crash can lose the only copy of the failed event. If it writes the DLQ but never commits the source offset, the same failure may be routed again; the DLQ sink should therefore be idempotent on (source topic, partition, offset, attempt).
For a Kafka-only source-to-DLQ route, use a Kafka transaction to publish the DLQ record and commit the source offset together. For an external incident store or ticketing system, Kafka cannot make the two systems atomic; use an idempotency key and reconcile unknown outcomes. Kafka delivery guarantees explains why “exactly once” must name its boundary.
try {
process(record); // The business effect must be idempotent.
markProcessed(record); // Store the event ID with the effect when needed.
consumer.commitSync(offsetAfter(record));
} catch (PermanentRecordException error) {
ProducerRecord<String, DlqEnvelope> dlq = toDlq(record, error);
producer.send(dlq).get(); // Do not commit until the DLQ write is acknowledged.
consumer.commitSync(offsetAfter(record));
} catch (TransientDependencyException error) {
pauseOrRetry(record); // Leave the source offset behind while the retry budget remains.
}
This simple example makes the ordering visible but is not an atomic transaction between the producer and consumer. A production implementation should use sendOffsetsToTransaction for a Kafka-to-Kafka DLQ flow, handle producer fencing, and make replay idempotent. If the DLQ write fails, keeping the source offset uncommitted is safer than skipping the record.
When a record cannot be deserialized, the normal value deserializer may throw before application code sees it. Configure an error-handling deserializer or framework-level record recoverer that retains raw bytes and source metadata. Never commit an unreadable record without preserving a forensic copy unless the data owner explicitly accepts loss.
Inspecting and Replaying Messages
Replay is a controlled data operation, not a button that moves every DLQ record back to the source. Use this runbook:
- Alert the owning team and stop automatic replay while the cause is unknown.
- Group failures by exception, application version, schema version, source partition, and event type.
- Confirm the consumer fix with a representative copy in a test topic.
- Check the target sink’s idempotency, ordering, rate limit, and downstream capacity.
- Select a bounded DLQ range and preserve the original event ID and key.
- Replay to a versioned source or controlled retry topic, not directly to production blindly.
- Monitor source lag, duplicate effects, retry rate, and remaining DLQ age.
- Mark each envelope as replayed, failed again, or intentionally discarded with an audit record.
# Inspect DLQ records without changing the source group.
kafka-console-consumer.sh \
--bootstrap-server kafka:9092 \
--topic orders.inventory.dlq \
--from-beginning \
--property print.headers=true \
--max-messages 20
# Describe the replay group before a bounded replay.
kafka-consumer-groups.sh \
--bootstrap-server kafka:9092 \
--describe --group inventory-dlq-replay
Replay should be rate-limited. If the original failure was a database outage, replaying the entire backlog at once can recreate the outage. Use a separate replay consumer group, a canary batch, and a circuit breaker that pauses when downstream latency or error rate crosses its budget.
Preserve ordering deliberately. If OrderCreated is in the DLQ while a later OrderCancelled already succeeded, replaying only the failed event may apply an invalid transition. Use the event version or aggregate sequence to reject stale effects, replay a complete aggregate range, or route the event to manual reconciliation. Kafka ordering guarantees covers why a DLQ does not remove the original partition-order contract.
Retention, Security and Monitoring
The DLQ deserves a stronger operational contract than an ordinary transient retry topic:
- Retention: keep failures long enough for investigation, then archive or expire them intentionally.
- Durability: use replication,
acks=all, and suitablemin.insync.replicas; the failure record may be the only recovery copy. - Access: allow the service to write and a narrow operations group to read or replay; separate replay permission from inspection permission.
- Schema: version the envelope and retain raw bytes when deserialization is part of the failure.
- Metrics: track DLQ ingress, count, oldest age, bytes, failure categories, attempts, replay rate, and replay success.
- Alerts: alert on critical records, sudden rate increases, oldest age over the remediation SLO, and replay loops.
Create a dashboard by source topic, consumer group, error class, schema version, and owner. A total DLQ count hides a small but critical payment stream behind a large analytics stream. A DLQ with zero records can also be unhealthy if writes are failing or the consumer is silently dropping records, so correlate it with source commits and consumer errors.
Production Example
An inventory consumer reads orders.v1. A database timeout is retried in place three times with jitter. A schema validation error is permanent and is written to orders.inventory.dlq with the original bytes, key, source offset, schema version, and application version. The DLQ producer uses acks=all; the source offset is committed only after the DLQ write succeeds.
The on-call sees DLQ ingress grouped by a newly deployed schema version and pauses replay. The schema is corrected, a canary replay of 20 events succeeds, and a separate replay group drains the backlog at 500 records per second while inventory database utilization stays below its SLO. The inbox table’s event ID makes a crash-safe replay harmless. Each envelope is marked REPLAYED or FAILED_AGAIN, and the final report links the deployment, root cause, and affected offsets.
Interview Questions
1. What is a Kafka dead letter queue?
A Kafka DLQ is usually a dedicated topic where a consumer, connector, or stream processor stores records it cannot process on the normal path. It is an application-level failure boundary for standard consumers, not a guarantee that Kafka automatically moves failed records. A good design includes error metadata, ownership, retention, alerting, and a replay process.
2. When should a message go to a DLQ instead of being retried?
Route it after a bounded retry budget when the failure is permanent, the payload is poisonous, or a retryable dependency has exceeded its time budget. Invalid schema, malformed bytes, and deterministic validation errors should not be retried forever. Transient outages usually deserve backoff first so the DLQ does not hide a recoverable incident.
3. What metadata belongs in a DLQ record?
Keep the original topic, partition, offset, key, timestamp, event ID, consumer group, application version, attempt count, failure category, exception class, trace ID, and replay status. Preserve raw bytes when deserialization may be the problem, but redact sensitive values in logs and restrict topic access. Without source coordinates, operators cannot diagnose or replay confidently.
4. How do you avoid losing a message while writing to the DLQ?
Write the DLQ record successfully before committing the source offset. For a Kafka-to-Kafka route, publish the DLQ record and source offset in one Kafka transaction when the client and topology support it. For an external sink, use an idempotency key and reconciliation because Kafka cannot make that external write atomic.
5. How should DLQ messages be replayed?
Fix and test the underlying cause first, then replay a bounded canary through a separate group or controlled retry topic. Verify sink idempotency, ordering, rate limits, and downstream capacity before increasing the rate. Preserve the original event ID and record an audit status so a replay does not become an untraceable second delivery.
6. Can a DLQ solve a hot partition or high consumer lag?
It can remove a permanent poison record that blocks a partition, but it cannot fix slow valid work, insufficient partitions, or a saturated database. I classify the failure, inspect per-partition lag, and route only records that meet the DLQ policy. Otherwise the DLQ merely moves the backlog and hides the real capacity problem.
Conclusion
A Kafka dead letter queue is a durable quarantine and recovery boundary, not a permanent graveyard. Classify failures, retry transient errors with a budget, preserve source metadata, commit only after the DLQ write is durable, and replay through an idempotent, rate-limited path. The strongest design makes ownership, retention, security, ordering, and success criteria explicit before the first poison message arrives.
References
- Confluent: Apache Kafka Dead Letter Queue
- Apache Kafka KIP-1034: Dead Letter Queue in Kafka Streams
- Apache Kafka KIP-1191: Dead-Letter Queues for Share Groups
YouTube Videos
- “Best practices to Retry Messages from Dead Letter Queue in Kafka” - Vlogize [https://www.youtube.com/watch?v=V1LVppBY8oc]