Kafka Architecture: Producers, Brokers, Topics and Offsets

Kafka is often introduced as a message queue, but that shortcut hides the decisions that matter in production. A Kafka topic is a durable, replicated log split into partitions. Producers append records, consumers read at their own positions, and a consumer group shares partitions for parallel work. That model enables high throughput and replay, but it also makes keys, offsets, retention, lag, and failure recovery part of application design.

This Kafka architecture guide builds on queue vs pub/sub and message queue fundamentals. Read it before the next partition-focused lesson. The Backend Interview Mastery series provides the full path, while the Distributed Systems tag connects the surrounding reliability concepts.

Table of Contents

Kafka as a Distributed Log

Kafka stores records in append-only logs. A record has a topic, partition, offset, timestamp, optional key, and value. Kafka does not remove a record merely because one consumer finished it. Instead, it retains records according to topic policy, and each consumer group records how far it has read. That separation lets one group drive a live service while another replays the same history to rebuild analytics.

This is the architectural difference behind Kafka’s common use in event streaming, change data capture, audit pipelines, and data integration. It can also support asynchronous work, but it is not a drop-in replacement for every short-lived task queue. Its throughput and replay capabilities come with operational responsibilities: disk capacity, partition design, schema compatibility, and a consumer strategy for duplicates.

Core Components

Modern Kafka clusters use KRaft-based metadata quorum controllers rather than requiring ZooKeeper. Application clients bootstrap from one or more brokers, request metadata, and then communicate with the leader for the needed partition. That means a load balancer is not secretly routing every record through one central Kafka process; the client uses cluster metadata to reach the right broker.

The Produce and Consume Flow

flowchart TD
    A[Order service producer] --> B[Kafka metadata lookup]
    B --> C[Leader broker for partition]
    C --> D[Replica followers]
    C --> E[Durable topic partition]
    E --> F[Consumer group]
    F --> G[Inventory consumer]
    G --> H[Commit processed offset]

1. The producer serializes an event, chooses a topic and usually a key.
2. Kafka metadata identifies the leader broker for the selected partition.
3. The producer sends the record to that leader; the configured acknowledgement level determines when the send is considered successful.
4. Follower replicas replicate the leader’s log under Kafka’s replication rules.
5. A consumer fetches records from its assigned partition, processes them, and commits an offset after its chosen durability boundary.

Batching makes Kafka efficient. Producers accumulate records for a partition before sending them, and consumers fetch sequential ranges. The trade-off is latency: larger batches and longer linger intervals improve throughput but can delay a low-volume event. Tune from a measured latency and throughput target, not a copied configuration.

Topics, Partitions, and Ordering

A topic is split into partitions so Kafka can store and read data in parallel. Each partition is an ordered log, but Kafka does not promise one total order across the topic. If two records land in different partitions, their relative processing order is not meaningful.

Keys are how applications deliberately preserve per-entity order. For example, use orderId as the key for order.created, order.paid, and order.cancelled. Kafka’s partitioner maps the same key to the same partition while the topic’s partition count and partitioning strategy remain compatible. A consumer assigned that partition reads those records in append order.

flowchart TD
    A[order 42 event] --> B[Key: order-42]
    B --> C[Topic: orders]
    C --> D[Partition 2]
    E[order 99 event] --> F[Key: order-99]
    F --> C
    C --> G[Partition 0]
    D --> H[One group member]
    G --> I[Another group member]

Do not use a random key when sequence matters, and do not force all records into one partition merely to claim ordering. One partition limits a consumer group to one active consumer for that work and makes a hot partition. The next article examines keys, partition count, and parallelism in more depth.

Brokers, Replicas, and Leaders

Kafka replicates each partition across brokers. One replica is the leader, and clients normally produce to and consume from that leader. Follower replicas copy the log. If a broker fails, Kafka can elect an in-sync replica as a new leader, subject to the topic configuration and cluster health.

Replication factor and minimum in-sync replicas protect different parts of the failure story. A factor of three provides copies on three brokers. A producer using acks=all asks the leader to wait for all currently required in-sync replicas before acknowledging. Setting min.insync.replicas=2 with replication factor three can reject writes when fewer than two replicas are in sync rather than quietly acknowledging a write with weaker durability.

That improves safety but lowers write availability during replica trouble. It is a conscious trade-off: for a payment-state event, rejecting a write may be safer than accepting a record that could disappear in a leader failure. For disposable telemetry, a team may choose a different availability and loss posture. Tie the setting to a recovery objective, then test broker loss and slow-replica behavior.

Consumer Groups and Offsets

Every consumer group has its own position for each consumed partition. If a topic has six partitions, up to six consumers in one group can actively consume them in parallel. A seventh consumer is idle for that topic. A second group can independently read all six partitions because it has a separate purpose and separate offsets.

Offsets are positions, not acknowledgements with universal meaning. The typical safe sequence is process a record, durably apply the local effect, then commit the offset. If the process crashes before commit, Kafka may redeliver the record. If it commits first and crashes before the effect, the group can skip work. The first choice favors at-least-once processing; the second risks loss.

flowchart TD
    A[Fetch record at offset 142] --> B[Apply local database effect]
    B --> C{Transaction succeeded?}
    C -->|No| D[Do not commit; retry or DLQ]
    C -->|Yes| E[Commit offset 143]
    E --> F[Fetch next record]

During a rebalance, partition assignments move between group members. Consumer code must stop processing revoked partitions before another member begins them, and it must tolerate records near the last committed offset appearing again. Long, blocking processing can make rebalances disruptive; use bounded processing, appropriate poll intervals, and a design that can resume safely.

Delivery Guarantees and Idempotency

Kafka can provide strong guarantees within carefully defined boundaries, but “exactly once” is not a substitute for modeling the database or external system your consumer touches. Idempotent producers can avoid duplicate writes caused by producer retries, and Kafka transactions can atomically write records and commit consumed offsets for a Kafka-to-Kafka pipeline.

When a consumer writes to PostgreSQL, charges a card, or invokes an email provider, that external effect is outside a Kafka transaction. Use a durable deduplication table, an idempotency key, an outbox, or an explicit state machine. The queue vs pub/sub guide shows a practical database uniqueness boundary for duplicate events.

For normal at-least-once consumers, commit only after the durable local effect succeeds. Make a poison-message policy explicit: retry transient conditions with backoff, send permanent failures to a dead-letter topic with error metadata, and alert on it. Never repeatedly retry an invalid payload at full speed and call that reliability.

Retention, Replay, and Compaction

Retention defines how long Kafka keeps records independent of consumer progress. Time- or size-based retention is useful when consumers need a replay window for recovery, audits, or new derived views. It also consumes disk, affects recovery time, and creates data-governance obligations.

Log compaction is different. A compacted topic retains the latest record for each key eventually, while still allowing consumers to observe changes in order. It fits current-state streams such as account preferences or product availability. It does not mean every historical record remains forever, and consumers must handle tombstone records used to represent deletions.

Choose topic policy from the contract. An immutable order-event stream may need a fixed audit retention. A user-profile projection may need compaction plus privacy-aware deletion procedures. Do not put unrelated event types with different retention or access requirements into one broad topic merely to reduce topic count.

Lag, Backpressure, and Operations

Consumer lag is the gap between a partition’s latest offset and a group’s committed offset. It is a signal, not automatically an outage. A planned replay creates lag; rapidly growing lag with increasing oldest-event age means consumers cannot keep up or are failing.

Monitor per-topic and per-partition throughput, consumer lag, oldest unprocessed record age, rebalance frequency, under-replicated partitions, offline partitions, produce latency, rejected writes, disk usage, and consumer error/DLQ rates. Per-partition views reveal hot keys that a topic-level average hides.

Backpressure starts before the disk fills. Limit producer retries and request timeouts, size consumer concurrency to partitions and downstream capacity, and pause or shed noncritical producers when the downstream system is unavailable. A database can be damaged by an eager consumer group that drains Kafka faster than the database can accept writes. Kafka decouples rates; it does not remove capacity limits.

A Production Order Event Example

An order service writes an order row and an outbox row in one database transaction. An outbox relay publishes an order.created.v1 record with orderId as the Kafka key. The topic has a replication factor appropriate for the data, a defined retention period, and a schema compatibility policy.

The inventory group receives one copy and reserves stock. Its processed_events table makes a repeated event harmless. The analytics group receives another copy and can replay a week of records to rebuild a report. A notification group delivers an email with a provider idempotency key. Each group owns its offset and its failure handling; the order service does not synchronously coordinate their work.

The practical payoff is isolation. A stalled analytics consumer grows only its own lag. It does not block inventory or checkout. The cost is operational discipline: monitor all groups, evolve the event schema compatibly, retain enough history for recovery, and never assume a committed Kafka offset proves an external email or payment succeeded.

Interview Questions

1. What is the difference between a topic and a partition?

A topic is the logical event stream used for naming, authorization, retention, and schema governance. A partition is one ordered shard of that topic’s log. I choose partitions to provide enough producer and consumer parallelism, then use keys to keep related events ordered within one shard.

2. What does an offset represent?

An offset is the position of a record within one partition. It is not a global timestamp, a topic-wide sequence, or a customer-facing event ID. A consumer group commits offsets to record its recovery position, which lets it resume or replay according to its chosen processing boundary.

3. Why can Kafka deliver duplicate messages?

A producer can time out after the broker accepted a record, or a consumer can complete work and crash before committing its offset. Retrying is safer than assuming the result was lost, so duplicates are a normal possibility. I use idempotent producer settings where appropriate and make the consumer’s business effect idempotent with a durable key.

4. What happens when consumers outnumber partitions?

Within a consumer group, at most one consumer owns a partition at a time. Extra consumers sit idle for that topic because assigning two consumers to the same partition would break the group’s ordered work distribution. More throughput may require more partitions, but increasing partitions changes key distribution and does not fix a single hot key.

5. How would you investigate rising consumer lag?

I first split lag by partition and compare its growth rate with consumer throughput and oldest-record age. Then I check consumer errors, downstream database latency, rebalance events, hot keys, and under-replicated or slow broker partitions. I scale only after locating the bottleneck: extra consumers cannot help when there are no unassigned partitions or when the downstream dependency is already saturated.

Conclusion

Kafka architecture is a set of explicit contracts: partition-local order, replica-backed durability, per-group offsets, retained history, and bounded recovery from failure. Producers, brokers, topics, partitions, and consumers are simple individually. The quality of a production design comes from choosing keys, acknowledgement settings, retention, idempotency, and observability that match the business invariant.

References

1. Apache Kafka: Introduction
2. Apache Kafka: Design
3. Apache Kafka: Producer Configuration
4. Confluent: Kafka Concepts

YouTube Videos

1. Apache Kafka Architecture Explained: Topics, Partitions, Offsets, Producer and Consumer