Kafka Partitions: Ordering, Keys and Parallelism Explained

A checkout event stream can look perfectly healthy until one popular customer, seller, or order becomes a hot key. If every event for that key lands on one partition, correctness is preserved but one partition limits throughput. If the team removes the key to spread traffic, events can be processed out of order and a refund may arrive before its payment. Kafka partitions are the boundary where that trade-off becomes concrete.

This guide explains how to choose keys, preserve the ordering that matters, and size consumer parallelism without treating partition count as a harmless tuning knob. Start with queue vs pub/sub for the delivery model, then read Kafka architecture for the producers, brokers, consumer groups, and offsets around this decision.

Table of Contents

What a Kafka Partition Is

A Kafka topic is a named stream of records. Kafka divides that stream into partitions: independent, append-only logs stored and replicated by brokers. Each record has an offset that identifies its position inside one partition. An offset is not a global sequence number for the topic, so offset 42 in partition 0 has no ordering relationship with offset 42 in partition 3.

Partitions let Kafka distribute storage, network traffic, and consumer work. They are not merely folders under a topic. A producer chooses a partition for each record, a leader broker appends it, replicas copy it, and a consumer group assigns each partition to at most one active member at a time. This makes a partition the unit of both ordering and consumer-group parallelism.

flowchart TD
    A[Producer] --> B{Partitioner}
    B -->|customer-42 key| P0[Partition 0 ordered log]
    B -->|customer-91 key| P1[Partition 1 ordered log]
    B -->|customer-18 key| P2[Partition 2 ordered log]
    P0 --> C0[Consumer A]
    P1 --> C1[Consumer B]
    P2 --> C2[Consumer C]
    C0 --> D[Consumer group]
    C1 --> D
    C2 --> D

For one consumer group, Kafka will not have two active consumers read the same partition concurrently. That constraint prevents group members from racing through a single ordered log. A different consumer group can independently read every partition from its own offsets, which is how one topic can feed billing, search indexing, analytics, and notifications without those systems competing for messages.

Ordering Is Per Partition, Not Per Topic

Kafka guarantees the append order of records in a single partition. It does not guarantee a total order across partitions. This distinction is the most important answer in a Kafka partition interview question.

Consider a payment service that emits PaymentAuthorized and PaymentRefunded for order o-123. If both records use o-123 as their key, the partitioner routes them to the same partition and a single consumer-group member processes their log position in order. If one producer uses an order key and another uses no key, the records can land on different partitions. Their relative arrival and processing order is then undefined.

Ordering also depends on the producer and consumer behavior around Kafka. Retries can reorder records when idempotence is disabled or in-flight requests are configured unsafely. A consumer can preserve partition order while handing records to an unconstrained worker pool that finishes them out of order. Say precisely what boundary is ordered: Kafka log append, consumption, and business side effects are separate concerns.

Choosing a Message Key

A key should represent the smallest business entity whose events must be serialized. Common choices include orderId for an order lifecycle, customerId for customer state, or accountId for a balance stream. A good key has three properties:

1. All events requiring relative order use the same key and serialization.
2. The key has enough distinct values to spread normal traffic across partitions.
3. The key remains stable over the event lifecycle.

The default Kafka producer partitioner hashes a non-null key and maps it to a partition. A stable key therefore keeps its records together while allowing different keys to spread. A null key is useful only when no relationship needs ordering; producers may distribute those records for balance, but a later record for the same entity has no placement guarantee.

ProducerRecord<String, OrderEvent> record = new ProducerRecord<>(
    "order-events",
    event.orderId(), // Every event for this order uses the same partitioning key.
    event
);

producer.send(record, (metadata, error) -> {
    if (error != null) {
        // Do not acknowledge the business action until the delivery policy decides.
        throw new RuntimeException("Kafka publish failed", error);
    }
    metrics.recordPartition(metadata.partition());
});

Do not key by a field merely because it is available. Keying every event by tenantId preserves tenant order, but one large tenant can become a permanent hot partition. Keying by a random UUID balances load, but destroys the order needed by an order-state consumer. The right key follows the invariant, not the database schema.

A Practical Key Decision

For inventory reservations, productId may preserve updates to one stock counter, but a flash-sale product becomes a hot key. The safer design can be to make the database reservation conditional and use Kafka for asynchronous propagation, rather than pretending more partitions can serialize one inventory item faster. For customer notifications, customerId is often a good key because it preserves a customer’s sequence while distributing many customers. This is the same contention-versus-correctness decision explored in optimistic versus pessimistic locking, but Kafka moves the serialization boundary into an event log.

Partitions Set Consumer Parallelism

Within one consumer group, the maximum number of active consumers for a single topic is its partition count. A topic with six partitions can give work to at most six consumer instances in that group. Starting ten instances does not make it ten times faster; four members will be idle for that topic, though they may process other assigned topics.

flowchart TD
    P0[Partition 0] --> C1[Consumer 1]
    P1[Partition 1] --> C1
    P2[Partition 2] --> C2[Consumer 2]
    P3[Partition 3] --> C2
    P4[Partition 4] --> C3[Consumer 3]
    P5[Partition 5] --> C3
    C4[Consumer 4] --> I[Idle: no partition assignment]

The relationship is a ceiling, not a throughput promise. A consumer may handle several partitions, so six partitions and three consumers can work well if processing cost is balanced. Adding consumers reduces work per member only until each partition has an owner. Adding partitions helps only when work can distribute across keys and brokers have enough disk, CPU, and network capacity.

For a multi-topic consumer, its theoretical maximum is the total partitions of the subscribed topics, but uneven workloads still matter. Measure per-partition lag and processing time rather than relying on group-wide lag alone. One hot partition can keep customer-facing events behind while the aggregate lag looks acceptable.

How to Size a Topic

Start from measurable throughput and recovery requirements, then leave room for growth. Suppose a service needs to consume 24,000 records per second. A production consumer instance, after deserialization and its downstream calls, sustains 3,000 records per second at a safe latency. The raw calculation is:

Required active consumers = 24,000 / 3,000 = 8

Choose more than eight partitions when balanced keys, expected growth, maintenance headroom, and recovery speed justify it. For example, 12 partitions can run eight consumers normally, allow a failed member’s work to move, and support modest growth. It is not automatically better than eight: every partition consumes broker files, replica traffic, metadata, leader-election work, and client connections.

Avoid copying an arbitrary partition count from another team. Their message size, key cardinality, retention period, replication factor, consumer work, and recovery objective may be completely different. Capacity planning should use observed producer throughput and load tests, then be revisited as traffic changes.

Hot Keys and Uneven Partitions

Hashing distributes keys, not load. If a celebrity seller produces half of all marketplace events, every record for that seller still goes to one partition. Adding 100 partitions does not split that seller’s ordered stream. This is a hot-key problem, not a partition-count problem.

First confirm the invariant. If only per-order order matters, key by orderId instead of sellerId. If one entity truly needs a strict sequence, one partition owner is the cost of that semantics. You can optimize the consumer, batch independent work after a safe boundary, or redesign the business workflow, but you cannot keep a total sequence for one key while processing it concurrently without coordination.

If an event can be split safely, add an explicit sub-key only after proving that operations across sub-keys do not require order. For example, analytics events may use customerId + eventType or a random key because they are independently aggregatable. Never add a random suffix to a payment, balance, or inventory key just to flatten a dashboard: it changes correctness, not only performance.

Changing Partition Counts Safely

Increasing a topic’s partition count is operationally possible, but it changes the hash-to-partition mapping for many keys. Existing records stay in their original partitions; future records for the same key can map to a new partition. That means a consumer may see newer records from the new partition before it finishes older records from the old one. Do not promise end-to-end per-key ordering across that change without a migration plan.

Before changing the count:

1. Confirm whether per-key ordering across historical and new records is a business requirement.
2. Check broker capacity, replica placement, partition leader balance, and controller overhead.
3. Test producers, consumers, monitoring, and disaster recovery with the proposed topology.
4. Plan how consumers handle records from old and new partitions, including idempotency and version checks.
5. Roll out during a controlled window and monitor per-partition skew, lag, rebalance duration, and error rates.

For a strict ordering domain, a new topic with a deliberate keying and migration strategy can be safer than changing an active topic in place. Consumers can drain the old topic, use a versioned event sequence, or route new entities to the new topic. The cost is operational complexity; the benefit is making the order boundary explicit instead of silently weakening it.

Producer and Consumer Configuration

The key chooses the partition, but delivery configuration protects the sequence. Use idempotent production for important ordered streams, retain the producer defaults that keep in-flight behavior safe, and treat retries as part of the ordering design. Kafka’s ordering guarantee is meaningful only for records the producer successfully appends in the intended sequence.

# Producer: let Kafka deduplicate retries for the same producer session.
enable.idempotence=true
acks=all

# Consumer: process one partition's records in poll order before committing.
enable.auto.commit=false
max.poll.records=100

Manual commits do not make a consumer exactly once. Commit only after the side effect is durable, and make the downstream write idempotent with an event ID or version. If processing a record fails, decide whether to retry, pause, send it to a dead-letter path, or stop the partition. Skipping it without an explicit policy can violate the very order the key was chosen to preserve. See idempotency in REST APIs for the same principle at an API boundary.

Production Checklist

• Define the exact entity that needs order and use its stable ID as the key.
• Test that every producer uses the same topic, key serialization, and partitioning policy.
• Monitor records, bytes, lag, consumer processing time, and error rate by partition, not only by topic.
• Size partitions from peak load, measured consumer capacity, recovery time, and broker overhead.
• Use idempotent producers and idempotent side effects; ordering does not prevent duplicate delivery after failures.
• Treat a partition-count increase as a data-ordering change and rehearse it before production.
• Investigate null keys, hot keys, and a large max-to-median partition-lag ratio early.

For the broader failure-handling path, connect this design to database deadlocks and the upcoming consumer-group and offset topics in this series. In each case, progress comes from defining an ownership boundary, observing contention, and retrying only operations that are safe to repeat.

Interview Questions

1. Does Kafka guarantee ordering for a topic?

Kafka guarantees order within a partition, not across a whole multi-partition topic. I first ask what entity requires order, key every event for that entity consistently, and explain that the entity’s stream will be serialized by one partition owner in a consumer group.

2. What happens if there are more consumers than partitions?

For that topic in one consumer group, consumers without a partition assignment are idle. They may still provide failover capacity or process another subscribed topic, but they do not increase the active parallelism of this topic. The partition count is the upper bound.

3. Can I add partitions whenever throughput grows?

I can add them operationally, but I do not treat it as free. Hash mapping changes, so future records for an existing key can move while historical records remain in the old partition. I verify whether that breaks the required sequence and use a controlled migration if it does.

4. How do you handle a hot Kafka key?

I confirm whether its strict order is required. If it is, I optimize or redesign that one serialized workflow; more partitions do not split it. If the work can be independent, I choose a narrower key or an explicitly safe sub-key. Randomizing a correctness key is not a valid fix.

5. Does using a key make processing exactly once?

No. The key controls placement and per-partition order. A failure can still cause a consumer to reprocess a record after an uncommitted offset, and an external side effect can be repeated. I combine the key with idempotent production, deliberate commits, and idempotent downstream writes.

Conclusion

Kafka partitions turn abstract delivery requirements into concrete engineering limits. They provide ordered logs and parallelism, but only at partition scope. Choose a key from the business invariant, size partitions from measured capacity and recovery needs, watch for skew, and treat topology changes as ordering changes. That is more durable than adding partitions until a lag graph looks better.

References

1. Apache Kafka Introduction
2. Apache Kafka Producer Configuration
3. Apache Kafka Consumer Configuration
4. Apache Kafka Operations

YouTube Videos

1. Kafka Partitions and Consumer Groups