
Replicated workers often need one process to assign sequence numbers, accept writes, or schedule a job. Leader election chooses that owner and replaces it after failure without allowing two leaders to act at once. Continue from distributed locks, quorum reads and writes, and consistent hashing.
Table of Contents
Open Table of Contents
Why Elect a Leader
Leader-based coordination simplifies ordering and ownership. Followers can serve reads or stand by for failover while the leader serializes writes. The cost is a failover pause, a coordination dependency, and the need to prove that an old leader cannot continue after its lease expires.
flowchart TD
A[Workers] --> B[Coordination store]
B --> C{Quorum grants term 12}
C --> D[Leader A serves writes]
C --> E[Followers replicate]
D -. heartbeat lost .-> F[Lease expires]
F --> G[New election term 13]
G --> H[Leader B with higher fencing token]
Election Protocol
Candidates request a monotonically increasing term or epoch from a coordination system. A quorum grants one candidate the lease, and the candidate renews it before expiry. Followers reject commands from older terms. A strong coordination store, such as etcd or ZooKeeper, is useful because its own writes are quorum-ordered.
Do not equate a process being alive with being allowed to lead. A paused process may miss heartbeats yet continue executing after a lease expires. The leader must attach its term to every mutation, and the authoritative storage layer must reject stale terms.
Split Brain and Fencing
Split brain occurs when an isolated old leader and a newly elected leader both believe they are active. Fencing tokens solve it: every election receives a larger token, and the resource accepts only commands with the highest token it has seen. If the resource cannot enforce fencing, lease-based leadership is only a best effort.
Use a monotonic clock for local deadlines, but do not rely on synchronized wall clocks for correctness. Persist the term, leader ID, expiry, and last renewal. Expose election churn and lease-renewal latency so operators can distinguish a real outage from a slow network.
Operational Design
Keep election traffic separate from business traffic, use randomized candidate backoff, and add a grace period before declaring a leader dead. During no-leader windows, reject writes with a retryable error rather than accepting conflicting ownership. On recovery, a former leader must rejoin as a follower and reload state.
Interview Questions
1. Why not let every replica accept writes?
Multi-leader writes require conflict resolution and can violate ordering or uniqueness. A leader serializes a subset of decisions, reducing application complexity. If availability is more important, multi-leader or leaderless designs are possible, but their conflict model must be explicit.
2. What causes split brain?
Network partitions, long garbage-collection pauses, or delayed heartbeats can make a healthy old leader appear dead. A new leader is elected while the old process continues. Quorum election plus fencing at the resource prevents both from making valid mutations.
3. What happens when the leader crashes after accepting a write?
The write must be durable before the acknowledgement or be replayable from a log. The new leader reads the committed log and resumes from the last safe term. If the acknowledgement was sent before durability, the API must document possible loss or use a stronger commit rule.
4. How do you choose an election timeout?
Set it above normal heartbeat latency plus a pause budget, then measure false elections. A shorter timeout fails over faster but increases churn during transient latency. A longer timeout improves stability but extends the unavailable window.
5. Is a lease enough for correctness?
No. A lease limits how long a coordinator should act, but a paused process may not stop exactly at expiry. Pair leases with fencing tokens enforced by the database, queue, or scheduler. The resource, not the process, must decide whether a command is current.
Conclusion
Leader election is safe only when election, expiry, and resource writes share a clear term model. Use quorum coordination, monotonic epochs, fencing tokens, and retryable no-leader behavior. Instrument churn and recovery so a rare partition does not become silent data corruption.
References
- etcd: Runtime reconfiguration and leader election
- Raft consensus algorithm
- Apache ZooKeeper: Recipes and solutions
YouTube Videos
- “Leader Election and Consensus in Distributed Systems” - Hussein Nasser [https://www.youtube.com/watch?v=Ibf7wKf8MqU]