
An order consumer charges a card, then crashes before committing its Kafka offset. On restart, Kafka delivers the order again. If the service commits first, a crash can lose the charge request entirely. At-most-once, at-least-once, and exactly-once are not labels to add to a diagram; they describe where a failure can leave a gap or duplicate.
This guide compares Kafka delivery guarantees and shows how producer acknowledgements, consumer commits, Kafka transactions, and idempotent external sinks fit together. Read Kafka offsets for recovery positions and Kafka ordering guarantees for the sequence boundary.
Table of Contents
Open Table of Contents
- What Delivery Semantics Mean
- At-Most-Once Delivery
- At-Least-Once Delivery
- Exactly-Once Semantics in Kafka
- External Systems Need Their Own Boundary
- Choosing a Production Contract
- Failure Timeline
- Production Example
- Interview Questions
- 1. What is the difference between at-most-once and at-least-once?
- 2. Does Kafka provide exactly-once delivery to a database?
- 3. How do you make at-least-once processing safe?
- 4. What does
read_committeddo? - 5. Why is committing before processing dangerous?
- 6. How should a service handle an uncertain payment timeout?
- Conclusion
- References
- YouTube Videos
What Delivery Semantics Mean
Delivery semantics describe whether a record can be lost or delivered more than once between a producer, Kafka, a consumer, and its effect. They do not automatically describe the business outcome. A Kafka-to-Kafka pipeline can achieve exactly-once processing inside Kafka while a payment provider still sees a retry unless the provider supports an idempotency key.
| Semantic | Can a record be lost? | Can a record be delivered twice? | Typical technique |
|---|---|---|---|
| At-most-once | Yes, after an early commit or acknowledgement | No from the consumer contract | Commit before processing; accept loss risk |
| At-least-once | No, if retention and retries hold | Yes, after effect-before-commit failure | Commit after durable effect; idempotent sink |
| Exactly-once | Not within the defined atomic boundary | Not to a read-committed Kafka consumer | Kafka transactions or a transactional application boundary |
The first question in an interview is “exactly once between which systems?” Define the input, output, offset, and side-effect boundary before choosing configuration. Otherwise a design can claim exactly once while a database, email provider, or payment API remains outside the transaction.
At-Most-Once Delivery
At-most-once commits the recovery position before applying the effect, or acknowledges a producer request without a durable acknowledgement policy. If the process crashes after the commit and before the effect, the record is skipped on restart. This minimizes duplicates and can reduce latency, but it accepts data loss.
flowchart TD
A[Poll record] --> B[Commit next offset]
B --> C[Apply side effect]
C --> D{Process crashes?}
D -->|Yes| E[Record may be lost]
D -->|No| F[One visible effect]
At-most-once is reasonable for telemetry where an occasional sample is less expensive than duplicate processing, or for a derived cache that can be rebuilt. It is a poor default for payments, inventory, audit events, and order state. A “no duplicate” dashboard is not proof of correctness if records disappeared before the handler ran.
At-Least-Once Delivery
At-least-once applies the durable effect first and commits the next offset afterward. A crash between those operations causes a replay, so the sink must make repeating the same event harmless. This is usually the most practical contract for business workflows because losing an event is more damaging than reprocessing one.
flowchart TD
A[Poll record] --> B[Apply idempotent durable effect]
B --> C{Effect committed?}
C -->|No| D[Retry without advancing offset]
C -->|Yes| E[Commit next offset]
E --> F{Crash before commit response?}
F -->|Yes| G[Replay is safe]
F -->|No| H[Continue]
An idempotent sink can use a unique eventId, a conditional state transition, an inbox table, or an API provider’s idempotency key. The offset remains a recovery pointer; the sink’s uniqueness or version rule is what turns a replay into a no-op.
for (ConsumerRecord<String, OrderEvent> record : records) {
// The database transaction rejects a previously applied eventId.
orderRepository.applyIfNew(record.value().eventId(), record.value());
}
// Commit only after every effect in this bounded batch is durable.
consumer.commitSync();
Batching lowers commit overhead but expands the replay window. If one record is permanently invalid, pause or retry the partition and route that record to a dead-letter topic with its original topic, partition, offset, key, and error. Skipping it silently can violate both the delivery and ordering contracts.
Exactly-Once Semantics in Kafka
Kafka transactions can atomically publish output records and commit the consumed offsets for a Kafka-to-Kafka pipeline. A downstream consumer configured with isolation.level=read_committed sees only committed transactional records. The transaction is a useful atomic boundary, not a universal promise that every external system was changed once.
producer.initTransactions();
producer.beginTransaction();
try {
producer.send(new ProducerRecord<>("order-enriched", key, enriched));
// The Kafka consumer group offset is committed into this same transaction.
producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
producer.commitTransaction();
} catch (ProducerFencedException | KafkaException error) {
producer.abortTransaction();
throw error;
}
The producer needs a stable transactional.id per active instance and a fencing strategy for zombies. Consumers must opt into read-committed isolation when they require transactional visibility. Transactions also add coordination and latency, so they should be applied where the atomic Kafka boundary pays for itself.
External Systems Need Their Own Boundary
Kafka cannot include an arbitrary PostgreSQL transaction, HTTP request, card charge, or email delivery in its transaction protocol. For those integrations, choose a pattern that makes uncertainty recoverable:
- Transactional outbox: write the business state and an event row in one database transaction; a relay publishes the event.
- Inbox or processed-events table: store the event ID and business effect in one transaction, then commit the Kafka offset.
- Provider idempotency key: reuse the same key on an uncertain HTTP retry and reconcile the provider result.
- State machine: record
pending,confirmed, orfailedand run a reconciliation job for unknown outcomes.
The idempotency pattern is often the practical answer when a payment call succeeds but the consumer times out. The service should not guess whether the call happened; it should retry with the same key or query the provider.
Choosing a Production Contract
Choose the weakest semantics that satisfy the business invariant, then make the boundary observable:
| Workload | Recommended contract | Why |
|---|---|---|
| Metrics and best-effort telemetry | At-most-once or lossy batching | Missing a sample is acceptable |
| Search indexing or cache updates | At-least-once plus upsert | Rebuild and duplicate-safe writes are available |
| Orders, inventory, audit trail | At-least-once plus idempotent sink | Avoid losing business events |
| Kafka stream-to-stream state update | Kafka exactly-once transaction | Input offset and Kafka output share a boundary |
| Payment or email provider | At-least-once plus provider idempotency | External call is outside Kafka’s transaction |
Measure duplicates, rejected records, transaction aborts, commit latency, retry age, and reconciliation backlog. If the contract is not visible in metrics, the team will discover its real semantics during an incident.
Failure Timeline
The two commit orderings make the trade-off concrete:
Effect -> Commit
crash before effect = no progress; retry
crash after effect = duplicate delivery; idempotency absorbs it
crash after commit = recovery continues
Commit -> Effect
crash before commit = retry
crash after commit = possible loss; effect may never run
Producer acknowledgements have a similar boundary. acks=all with idempotence reduces duplicate appends and protects acknowledged records, but an application timeout can still leave the final broker state uncertain. The producer should retry safely or reconcile using a stable event ID rather than blindly publishing a second business action.
Production Example
An order service writes the order state and an outbox row in PostgreSQL. A relay publishes the outbox event with an idempotent Kafka producer. The inventory consumer stores (consumer, eventId) in an inbox table in the same transaction as its stock update, then commits the Kafka offset. Analytics uses a separate consumer group and can replay from a historical offset without charging or reserving stock again.
If Kafka is unavailable after the order transaction, the outbox row remains pending. If inventory crashes after its database commit, the replay finds the inbox row and returns success. If a payment provider times out, the payment state remains pending and a reconciliation worker retries with the provider’s idempotency key. Each system has a boundary it can actually enforce.
Interview Questions
1. What is the difference between at-most-once and at-least-once?
At-most-once advances recovery before the effect and may lose a record, while at-least-once applies the effect before advancing recovery and may replay it. I choose based on whether loss or duplication is more dangerous. For business events, at-least-once plus an idempotent sink is usually the safer contract.
2. Does Kafka provide exactly-once delivery to a database?
Not by itself. Kafka transactions can bind Kafka input offsets to Kafka output records, but a database transaction is outside that protocol. I use an outbox, inbox, unique event record, or reconciliation state machine and state exactly which operations share the atomic boundary.
3. How do you make at-least-once processing safe?
Persist a stable event ID and apply the business mutation conditionally in the same transaction. Commit the Kafka offset only after that transaction succeeds. On replay, the uniqueness check or version guard returns the prior result instead of applying a second charge or reservation.
4. What does read_committed do?
It makes a Kafka consumer hide records from aborted transactions and expose only committed transactional writes. It does not make a database or HTTP side effect atomic with Kafka. The consumer still needs a sink strategy and a clear offset boundary.
5. Why is committing before processing dangerous?
A crash after the commit but before the effect causes Kafka to skip the record on restart. That is at-most-once behavior and can silently lose an order or audit event. It is appropriate only when the workload explicitly accepts loss.
6. How should a service handle an uncertain payment timeout?
It should retry or query using the same provider idempotency key, not generate a second charge request. Store a durable payment state such as pending and reconcile unknown outcomes. Kafka’s offset alone cannot tell whether the provider received the first request.
Conclusion
Delivery semantics are failure contracts, not marketing labels. At-most-once trades loss for fewer duplicates; at-least-once preserves events by requiring idempotent effects; Kafka exactly-once is powerful inside a Kafka transaction but stops at external system boundaries. Define the boundary, implement it explicitly, and measure the failures that remain.
References
- Apache Kafka: Design and Delivery Semantics
- Apache Kafka: Transactions
- Confluent: Message Delivery Guarantees
YouTube Videos
- “Kafka Delivery Semantics: At Most Once, At Least Once, Exactly Once” - Conduktor [https://www.youtube.com/watch?v=V0c0qAP7sWk]