
Two workers see the same scheduled job and both start sending invoices. A local mutex cannot coordinate processes on different hosts, so the system needs a shared ownership contract. Distributed locks can serialize work, but a lock that expires while its holder is still running can allow two owners unless the protected resource also verifies ownership.
This guide covers leases, fencing tokens, Redis, database locks, ZooKeeper-style coordination, and safer designs. Continue from race conditions, idempotency, and leader election.
Table of Contents
Open Table of Contents
Lock vs Lease
A lock grants one owner access until it explicitly releases. A lease grants ownership for a bounded time and requires renewal. Leases are safer than locks that can remain forever after a crash, but they introduce clock, renewal, and pause problems.
flowchart TD
A[Worker requests lease] --> B{Lease available?}
B -->|No| C[Back off and retry]
B -->|Yes| D[Grant token and expiry]
D --> E[Worker performs work]
E --> F[Renew before deadline]
F -->|Renew succeeds| E
F -->|Renew fails or pauses| G[Lease may expire]
G --> H[New worker receives higher token]
H --> I[Resource rejects stale token]
The lock service can decide who owns a lease, but it cannot stop a paused process from continuing its critical section. The resource being changed must reject stale owners or the lock is only advisory.
Failure Timeline
Worker A acquires lease token 41, then pauses for garbage collection. The lease expires and worker B acquires token 42. A resumes and writes to the database. Without a fencing check, A’s late write can corrupt the state even though the lock service correctly granted B ownership.
Store the token with the protected record and require WHERE fencing_token < 42 or an equivalent compare-and-set. The database becomes the final arbiter of stale ownership. A random lock value proves release ownership; a monotonically increasing fencing token protects the resource from delayed work.
Fencing Tokens
Acquire a unique token with the lease. Every write carries it, and the resource accepts only a token greater than or equal to the last accepted token for that logical job. Keep the check in the same transaction as the side effect when possible.
UPDATE jobs
SET status = 'RUNNING', fencing_token = :token
WHERE job_id = :job_id
AND fencing_token < :token
AND status IN ('READY', 'RUNNING');
If the update affects zero rows, the worker must stop; it no longer owns the job. This pattern is more reliable than trusting wall-clock expiry or a best-effort unlock.
Implementation Choices
| Mechanism | Strength | Main risk |
|---|---|---|
| Database row lock | Transactional with the data | Long transactions and contention |
| Redis lease | Low latency and simple TTL | Failover, pauses, and stale owners |
| ZooKeeper/etcd | Coordinated ephemeral ownership | Operational quorum dependency |
| Queue partition ownership | Work naturally serialized | Requires partition/key design |
Use a database conditional update or queue ownership when the business data already has a natural serialization point. Add a distributed lock only when the critical section is small, the failure contract is explicit, and the resource can fence stale work.
When Not to Use a Distributed Lock
Prefer idempotency keys, unique constraints, optimistic version checks, or a single-writer partition when they express the business rule directly. A lock does not make an external API call atomic, and holding it across a slow network call creates availability and lease-renewal risk. For scheduled jobs, duplicate execution plus idempotent effects is often safer than a large lock critical section.
Interview Questions
1. Why can a distributed lock expire while work continues?
The process can pause due to garbage collection, CPU starvation, network delay, or a lost renewal response. The lock service sees the lease expire and grants another owner while the paused process later resumes. Fencing tokens make the old work harmless at the resource.
2. What is a fencing token?
It is a monotonically increasing ownership number issued with a lease. Every protected write carries the token, and the resource rejects older tokens. This prevents delayed work from an expired owner from overwriting a newer owner’s state.
3. Is Redis SET NX EX enough for a lock?
It can implement a basic lease with an owner value and safe compare-and-delete release, but it does not by itself protect a resource from a paused stale owner. Add fencing or use a transactional primitive appropriate to the data. Understand the Redis deployment and failover assumptions before treating it as a correctness boundary.
4. How should a lock holder handle renewal failure?
Stop starting new side effects, attempt a safe handoff or rollback, and let the resource reject stale tokens. Do not continue indefinitely because the client believes it still owns the lease. Emit metrics for renew latency, expiry, and rejected stale writes.
5. When is a lock worse than idempotency?
When the operation can safely run twice and converge using a unique key or conditional update. Locks add coordination, expiry, and an availability dependency. Idempotent effects often let the system tolerate crashes without holding exclusive ownership across slow work.
Conclusion
Distributed locks coordinate intent, but leases expire and processes pause. Use short critical sections, explicit ownership, safe release, and fencing tokens at the protected resource. Prefer idempotency, conditional writes, queues, or single-writer partitioning when those mechanisms represent the business invariant more directly.
References
YouTube Videos
- “Distributed Locks and Leases Explained” - Distributed Systems Concepts [https://www.youtube.com/watch?v=d2z78guUR4g]