
Redis can be a cache, session store, queue, rate limiter, or lock service. When it fails, the right answer depends on which contract it serves. This scenario applies distributed caching, circuit breakers, and distributed locks.
The Interview Question
“Redis is unavailable for ten minutes. Should the application fail, fall back, or continue? Design the response for cache reads, sessions, and rate limits.”
Clarifying Questions
Ask which data is authoritative, whether sessions are stateless, the database’s spare capacity, and whether security controls may fail open. Assume product data is cacheable, login sessions use signed cookies plus Redis, and rate limiting must remain conservative.
The Naive Design and Why It Fails
Retrying Redis synchronously on every request turns a cache outage into connection and thread exhaustion. Letting all requests fall through can overload the database. Failing open for locks or rate limits can cause duplicate work or abuse.
Better Design
Classify each use. Cache misses can fall back to the system of record with a bounded concurrency gate. Session requests can revalidate a signed token or require re-login. Security-critical limits and locks fail closed or move to a durable alternative.
flowchart TD
A[Request] --> B{Redis operation}
B -->|Cache read| C[Bounded DB fallback]
B -->|Session| D[Signed token or re-auth]
B -->|Rate limit| E[Conservative fail closed]
B -->|Lock| F[Durable owner or reject]
C --> G[Serve degraded response]
Classify the Redis Contract Before Choosing a Fallback
The same outage has different consequences for different keys:
| Redis use | Is Redis authoritative? | Safe outage behavior |
|---|---|---|
| Product or configuration cache | No | Read from the database through a bounded gate; serve stale local data only with a freshness limit. |
| Short-lived session data | Sometimes | Validate a signed token locally, use a replicated session store, or require re-authentication for sensitive actions. |
| Rate-limit counters | Usually no, but security-sensitive | Apply a conservative local limit or reject the request; never let every caller through by accident. |
| Distributed locks | Depends on the invariant | Fail closed for payments, inventory, and leader ownership; use a durable owner and fencing token when possible. |
| Queue or stream | Often yes | Stop acknowledging new work, preserve the producer’s durable outbox, and replay after the broker recovers. |
This classification is the core interview decision. “Redis is down” is not enough information; the answer must name the business fact that would be lost or violated if the application continued.
Protecting the Fallback Path
The fallback must have its own budget. If normal traffic is 20,000 requests per second and Redis normally serves 95% of reads, a total outage can turn the database load from 1,000 to 20,000 reads per second. A semaphore or separate connection pool should admit only what the database can absorb, while the rest receives a fast, retryable response. Retries to Redis must also have a short deadline; a cache timeout should not hold an application thread for the duration of the user request.
try Redis with a 20 ms deadline
if hit: return the value
if timeout or error:
open the cache circuit briefly
acquire one fallback permit
if no permit: return a degraded or retryable response
read the authoritative store
populate Redis only after the outage is recovering
Do not refill every key as soon as the cluster returns. Use TTL jitter, request coalescing, and gradual warming. Otherwise thousands of concurrent misses can create a recovery stampede that takes down the database after the original Redis incident is over.
Recovery Runbook
First confirm whether the failure is a client-side connection problem, a primary failover, network reachability, memory pressure, or data corruption. Freeze risky configuration changes, record the circuit state and fallback rate, and verify that replicas agree before reopening traffic. Restore in stages: health checks, a small canary of cache reads, gradual key warming, then session, rate-limit, and lock traffic. Compare hit ratio, command latency, evictions, fallback QPS, and database saturation at each step.
If Redis held the only copy of a business fact, do not reconstruct it from guesses. Mark the affected workflow unknown, restore from persistence or an append-only event log, and reconcile with the authoritative external system. Availability is less important than silently granting access, charging twice, or losing a job.
Recovery and Failure Handling
Open a circuit after fast failures, cap fallback concurrency, and add jittered retries only in a small control path. Warm hot keys gradually after Redis returns; otherwise a synchronized refill causes another spike. Verify replication and persistence before declaring recovery.
Scaling and Trade-offs
If the database has 10,000 read QPS of spare capacity, the fallback gate must stay below that budget after accounting for normal traffic. A stale local cache can improve availability but consumes memory and complicates invalidation. High availability for Redis reduces risk but does not remove the need for graceful degradation.
Interview Follow-Ups
1. Should a cache outage make reads fail?
Not when the database is authoritative and can absorb bounded fallback traffic. Use a circuit breaker and concurrency limit to protect it. If data exists only in Redis, surface unavailability or restore from durable persistence.
2. What about Redis-backed sessions?
Prefer short-lived signed access tokens whose claims can be validated locally, with a revocation strategy for high-risk actions. If server-side sessions are mandatory, fail closed or route to a replicated session store rather than accepting unauthenticated requests.
3. How do you avoid a cache stampede on recovery?
Warm keys gradually, add request coalescing, and use TTL jitter. Keep fallback concurrency bounded until hit ratio stabilizes. Observe database load and queue age while reopening the circuit.
4. Can a lock fail open?
Only when duplicate execution is harmless and idempotent. For payments, inventory, or leader ownership, fail closed or use a fencing-capable alternative. A timeout acquiring a lock is not proof that no other worker holds it.
5. What do you alert on?
Redis errors and latency, connection exhaustion, fallback QPS, database saturation, circuit state, hit ratio, and recovery warm-up rate. Segment cache and critical-control traffic so a healthy cache does not hide a failing lock path.
Related Practice
Review backpressure, health checks, and traffic growth.
References
YouTube Videos
- “What Happens When Redis Goes Down?” - Hussein Nasser [https://www.youtube.com/watch?v=Qg4aPDZFbkA]