Skip to content
ADevGuide Logo ADevGuide
Go back

Distributed Locks: Leases, Failure Modes and Safe Use

By Pratik Bhuite | 9 min read

Hub: Java / Interview Fundamentals

Series: Backend Interview Mastery

Last verified: Sep 19, 2026

Part 51 of 88 in the Backend Interview Mastery

Key Takeaways

On this page
Reading Comfort:

Distributed locks coordinate workers with leases, expiration, and fencing tokens

Two workers see the same scheduled job and both start sending invoices. A local mutex cannot coordinate processes on different hosts, so the system needs a shared ownership contract. Distributed locks can serialize work, but a lock that expires while its holder is still running can allow two owners unless the protected resource also verifies ownership.

This guide covers leases, fencing tokens, Redis, database locks, ZooKeeper-style coordination, and safer designs. Continue from race conditions, idempotency, and leader election.

Table of Contents

Open Table of Contents

Lock vs Lease

A lock grants one owner access until it explicitly releases. A lease grants ownership for a bounded time and requires renewal. Leases are safer than locks that can remain forever after a crash, but they introduce clock, renewal, and pause problems.

flowchart TD
    A[Worker requests lease] --> B{Lease available?}
    B -->|No| C[Back off and retry]
    B -->|Yes| D[Grant token and expiry]
    D --> E[Worker performs work]
    E --> F[Renew before deadline]
    F -->|Renew succeeds| E
    F -->|Renew fails or pauses| G[Lease may expire]
    G --> H[New worker receives higher token]
    H --> I[Resource rejects stale token]

The lock service can decide who owns a lease, but it cannot stop a paused process from continuing its critical section. The resource being changed must reject stale owners or the lock is only advisory.

Failure Timeline

Worker A acquires lease token 41, then pauses for garbage collection. The lease expires and worker B acquires token 42. A resumes and writes to the database. Without a fencing check, A’s late write can corrupt the state even though the lock service correctly granted B ownership.

Store the token with the protected record and require WHERE fencing_token < 42 or an equivalent compare-and-set. The database becomes the final arbiter of stale ownership. A random lock value proves release ownership; a monotonically increasing fencing token protects the resource from delayed work.

Fencing Tokens

Acquire a unique token with the lease. Every write carries it, and the resource accepts only a token greater than or equal to the last accepted token for that logical job. Keep the check in the same transaction as the side effect when possible.

UPDATE jobs
SET status = 'RUNNING', fencing_token = :token
WHERE job_id = :job_id
  AND fencing_token < :token
  AND status IN ('READY', 'RUNNING');

If the update affects zero rows, the worker must stop; it no longer owns the job. This pattern is more reliable than trusting wall-clock expiry or a best-effort unlock.

Implementation Choices

MechanismStrengthMain risk
Database row lockTransactional with the dataLong transactions and contention
Redis leaseLow latency and simple TTLFailover, pauses, and stale owners
ZooKeeper/etcdCoordinated ephemeral ownershipOperational quorum dependency
Queue partition ownershipWork naturally serializedRequires partition/key design

Use a database conditional update or queue ownership when the business data already has a natural serialization point. Add a distributed lock only when the critical section is small, the failure contract is explicit, and the resource can fence stale work.

When Not to Use a Distributed Lock

Prefer idempotency keys, unique constraints, optimistic version checks, or a single-writer partition when they express the business rule directly. A lock does not make an external API call atomic, and holding it across a slow network call creates availability and lease-renewal risk. For scheduled jobs, duplicate execution plus idempotent effects is often safer than a large lock critical section.

Interview Questions

1. Why can a distributed lock expire while work continues?

The process can pause due to garbage collection, CPU starvation, network delay, or a lost renewal response. The lock service sees the lease expire and grants another owner while the paused process later resumes. Fencing tokens make the old work harmless at the resource.

2. What is a fencing token?

It is a monotonically increasing ownership number issued with a lease. Every protected write carries the token, and the resource rejects older tokens. This prevents delayed work from an expired owner from overwriting a newer owner’s state.

3. Is Redis SET NX EX enough for a lock?

It can implement a basic lease with an owner value and safe compare-and-delete release, but it does not by itself protect a resource from a paused stale owner. Add fencing or use a transactional primitive appropriate to the data. Understand the Redis deployment and failover assumptions before treating it as a correctness boundary.

4. How should a lock holder handle renewal failure?

Stop starting new side effects, attempt a safe handoff or rollback, and let the resource reject stale tokens. Do not continue indefinitely because the client believes it still owns the lease. Emit metrics for renew latency, expiry, and rejected stale writes.

5. When is a lock worse than idempotency?

When the operation can safely run twice and converge using a unique key or conditional update. Locks add coordination, expiry, and an availability dependency. Idempotent effects often let the system tolerate crashes without holding exclusive ownership across slow work.

Conclusion

Distributed locks coordinate intent, but leases expire and processes pause. Use short critical sections, explicit ownership, safe release, and fencing tokens at the protected resource. Prefer idempotency, conditional writes, queues, or single-writer partitioning when those mechanisms represent the business invariant more directly.

References

  1. Redis: Distributed locks
  2. Martin Kleppmann: How to do distributed locking
  3. etcd: Concurrency API

YouTube Videos

  1. “Distributed Locks and Leases Explained” - Distributed Systems Concepts [https://www.youtube.com/watch?v=d2z78guUR4g]

Share this post on:

Next in Series

Continue through the [object Object] with the next recommended article.

Related Posts

Keep Learning with New Posts

Subscribe through RSS and follow the project to get new series updates.

Was this guide helpful?

Share detailed feedback

Previous Post
Consistent Hashing: Caches, Sharding and Rebalancing
Next Post
Distributed Transactions: Why Local ACID Is Not Enough