Cache Stampede: Avalanche, Penetration and Prevention

At 50 requests per second, an expensive product query may be harmless. At 20,000 requests per second, one popular cache key expiring can turn a cache into an amplifier: every request misses, every application instance runs the same query, and the database queue grows faster than it can drain. The resulting outage is usually called a cache stampede, but that name hides two different failure modes that need different defenses.

This guide follows cache invalidation strategies. It separates a stampede on one hot key from a cache avalanche across many keys and cache penetration for keys that never exist. You will learn how to protect the origin with request coalescing, stale-while-revalidate, TTL jitter, negative caching, and a Bloom filter. Start with cache-aside if the read path itself is unfamiliar, then connect these safeguards to the operational limits in Redis architecture.

Table of Contents

Three Failures That Sound Alike

The terms are often mixed together in interviews. Naming the blast radius precisely is the first useful diagnosis.

A flash-sale product page demonstrates the distinction. If product:42 expires at noon and 5,000 requests race to rebuild it, that is a stampede. If one deployment gives 500,000 product keys the same 30-minute TTL and they all expire at noon, that is an avalanche; every individual product may then experience its own stampede. If bots request random IDs such as product:999999999 that have never existed, that is penetration. One Redis lock cannot solve all three.

The Backend Interview Guide is a useful way to frame the answer: state the failure condition, estimate the origin’s safe capacity, then choose controls that preserve the product’s freshness contract. A product title may be safe to serve 30 seconds stale. Inventory and price might not be. The correct solution therefore begins with the data, not with a fashionable cache primitive.

Why a Miss Storm Becomes an Outage

Assume a database can sustainably perform 1,000 of these product queries per second. The service receives 20,000 requests per second, usually with a 98% cache hit rate. The database handles about 400 cache-miss queries per second and stays healthy. When a hot key expires, a naive cache-aside implementation lets every concurrent request query the database while a replacement is being computed.

If that refill takes 200 ms, approximately 4,000 requests can arrive during the empty interval at 20,000 requests per second. They do not merely create 4,000 reads: they consume connections, queue behind locks, extend the refill latency, and cause retries. The longer refill interval then admits even more work. A 20-second connection timeout makes this feedback loop worse because old work remains in the queue after the request that created it is no longer useful.

flowchart TD
    A[Hot key expires] --> B[Concurrent cache misses]
    B --> C[Duplicate origin reads]
    C --> D[Origin queue grows]
    D --> E[Refill becomes slower]
    E --> B
    D --> F[Timeouts and retries]
    F --> B

The goal is not simply to restore the hit rate. It is to keep live origin work below a known budget while cache state is unavailable. That requires a short timeout at each dependency, a bounded number of refreshers, and a product decision for waiters: wait briefly, receive an acceptable stale response, or receive a deliberate overload response. Unbounded waiting is not graceful degradation.

Prevent Cache Stampedes for One Hot Key

Start with Local Request Coalescing

Within one application process, all callers for the same key can share one in-flight promise. This is usually the cheapest protection because no network round trip or distributed-lock lease is required. It is also incomplete in a fleet: 20 pods can still start 20 refreshes. Use local coalescing first, then add cross-process coordination only where the remaining origin work is unsafe.

const inFlight = new Map<string, Promise<Product | null>>();

async function loadOnce(key: string, load: () => Promise<Product | null>) {
  const existing = inFlight.get(key);
  if (existing) return existing;

  const task = load().finally(() => {
    // Remove the promise even when the origin fails so a later request can retry.
    inFlight.delete(key);
  });

  inFlight.set(key, task);
  return task;
}

The map needs key cardinality controls. Never let an attacker create an unbounded number of distinct in-flight entries. Normalize and authorize the key first, cap concurrent origin work globally, and use the caller’s deadline rather than letting a slow origin read live forever.

Use a Per-Key Lease Across Instances

When a hot key is shared across instances, one worker can acquire a short Redis lease before rebuilding it. The lock protects the origin from duplicate recomputation; it is not a correctness lock for payments, inventory writes, or other irreversible actions. The holder must use a unique token, a lease shorter than the request budget, and token-checked release. A plain DEL lock:key is unsafe because a late former owner can delete a newer owner’s lease.

import { randomUUID } from "node:crypto";

const releaseLease = `
  if redis.call("GET", KEYS[1]) == ARGV[1] then
    return redis.call("DEL", KEYS[1])
  end
  return 0
`;

async function refreshProduct(productId: string) {
  const cacheKey = `product:${productId}`;
  const lockKey = `lock:${cacheKey}`;
  const token = randomUUID();
  const acquired = await redis.set(lockKey, token, { NX: true, PX: 2_000 });

  if (acquired !== "OK") return null;

  try {
    const product = await productRepository.findById(productId);
    await redis.set(cacheKey, JSON.stringify(product), { EX: 60 });
    return product;
  } finally {
    await redis.eval(releaseLease, { keys: [lockKey], arguments: [token] });
  }
}

If the lease cannot be acquired, do not blindly spin until it expires. A waiter can poll once with jitter, return a stale value inside a hard freshness limit, or fail fast with a retriable response. Select that behavior per endpoint. The Redis documentation describes SET with NX and expiry for a basic lease pattern; its distributed-lock guidance also explains why ownership-safe release matters.

Serve Stale While One Request Revalidates

For data with a clear staleness budget, retain the value beyond its fresh window. At a soft expiry, one worker refreshes in the background while other readers receive the old value. At a hard expiry, stop serving it. This converts a sharp miss cliff into bounded staleness.

fresh for 60 seconds -> may serve stale for 30 seconds -> must reload or fail

For a news headline or catalog description, 30 seconds of staleness may be preferable to an origin outage. For an authorization decision, it may not be acceptable at all. HTTP caches expose the same idea through stale-while-revalidate; application caches need the same explicit freshness and hard-age rules. Do not call a response “eventually fresh” without specifying the maximum age a user can observe.

Refresh Before the Cliff for Very Hot Keys

The hottest keys may justify probabilistic early refresh: as a value approaches expiry, a small and increasing fraction of readers elect to refresh it. This spreads refresh work over time instead of waiting for a synchronized zero-TTL boundary. It is useful only after measuring a true hot-key pattern; early refresh spends more origin capacity by design. Pair it with coalescing so several early refresh attempts do not recreate the herd.

Prevent Cache Avalanches Across Many Keys

An avalanche is a fleet-level event. Common causes include identical TTLs assigned during an import, a broad FLUSH operation, cache-node loss, a bad deployment that changes key namespaces, and a traffic ramp that hits an empty cache. Per-key locking may reduce duplicate work for individual keys, but it cannot make the database absorb a million distinct misses.

Add TTL Jitter Deliberately

Instead of expiring every catalog object at exactly 30 minutes, choose a base TTL plus a bounded random offset. For example, a 30-minute base with up to five minutes of positive jitter distributes expiries across five minutes. The jitter must be large enough relative to request volume and origin capacity to matter; a random 100 ms offset does not protect a 10-minute import.

function ttlWithJitter(baseSeconds: number, jitterSeconds: number) {
  return baseSeconds + Math.floor(Math.random() * (jitterSeconds + 1));
}

await redis.set(cacheKey, payload, {
  EX: ttlWithJitter(1_800, 300),
});

Jitter changes when data is refreshed, not whether it is fresh enough. If every key must be invalidated immediately after a price update, use the targeted invalidation behavior from cache invalidation strategies and protect the subsequent refill wave with admission control or staged warming.

Warm and Ramp Instead of Flushing

Cache warming is safe only when it has a priority list and a rate limit. After a restart, warm the small set of keys that carry most request volume, observe database utilization and cache hit ratio, then increase the rate. Loading every key eagerly can create the same avalanche before real traffic arrives.

A practical rollout sequence is:

1. Keep the old key namespace readable during a short transition when possible.
2. Preload the measured top keys at a rate that leaves origin headroom.
3. Route a small percentage of traffic to the new namespace.
4. Increase traffic only while origin latency, connection-pool use, and error rate remain within thresholds.
5. Stop or roll back the ramp when the origin crosses its safe budget.

For cache-node failure, decide whether the service should bypass to the source of truth, serve stale, rate limit, or fail closed. The answer cannot be universal. The cache eviction policies guide helps distinguish expected evictions from an actual cache-tier failure; both can produce misses, but their operational response is different.

Prevent Cache Penetration for Missing Data

Cache penetration occurs when a request asks for an absent key and the application checks the database every time. It is common in bot traffic, malformed public URLs, enumeration attacks, and products whose records have been deleted. A conventional cache has no value to return, so it repeatedly forwards the load to the source of truth.

Validate and Negative-Cache Legitimate Absence

Reject malformed identifiers before the cache. A UUID parser, tenant authorization check, and route validation cost far less than a database lookup. For valid requests that truly find no record, cache a sentinel such as NOT_FOUND for a short TTL. The short TTL is important: a product created moments later must not remain invisible for an hour.

const missing = "__not_found__";

async function getProduct(id: string) {
  if (!isPublicProductId(id)) return null;

  const cached = await redis.get(`product:${id}`);
  if (cached === missing) return null;
  if (cached) return JSON.parse(cached) as Product;

  const product = await productRepository.findPublicById(id);
  if (!product) {
    await redis.set(`product:${id}`, missing, { EX: 30 });
    return null;
  }

  await redis.set(`product:${id}`, JSON.stringify(product), { EX: 300 });
  return product;
}

Negative caching is unsuitable when absence can change immediately or when exposing an absence leaks information. For example, do not cache a permission-denied response under a shared resource key. Put authorization in the key or check it before caching.

Filter Impossibilities with a Bloom Filter

A Bloom filter can answer “definitely not present” without querying the database. It may return a false positive, which means the application still checks the cache and database, but it must never treat a positive result as proof that a record exists. It is a load-shedding hint, not an authorization system or source of truth.

Populate the filter from the authoritative creation stream or a periodic rebuild, and monitor its false-positive rate and freshness. A filter that is not updated after a new record is created will cause a false negative at the application layer, so the write path needs careful ordering. For rapidly changing or small datasets, short negative caching and input validation are often simpler than introducing a probabilistic structure.

flowchart TD
    A[Request for product ID] --> B{Valid ID and authorized?}
    B -->|No| C[Reject before origin]
    B -->|Yes| D{Bloom filter says possible?}
    D -->|No| E[Return not found]
    D -->|Yes| F{Cache value?}
    F -->|Found| G[Return cached product]
    F -->|Missing| H[Query authoritative store]
    H -->|Absent| I[Short negative cache]
    H -->|Present| J[Cache product with jitter]

A Production Read Path

The controls work as layers rather than as alternatives. This sequence protects the common path and makes overload behavior explicit:

1. Validate the request and authorize the caller before creating cache keys.
2. Use a Bloom filter only when the key space is large, mostly absent, and the filter is maintained correctly.
3. Return a fresh cache value immediately.
4. Return bounded-stale data while a single local or distributed refresher revalidates it.
5. Coalesce duplicate refreshes, then apply a per-key lease only if multiple instances make the residual work unsafe.
6. Put an origin concurrency limit and deadline around refreshes; reject or degrade excess work instead of queuing it without limit.
7. Use negative caching for genuine absence, with a TTL tied to creation latency and correctness requirements.

The cache’s own availability deserves a separate plan. A Redis outage can make every request take the expensive fallback path at once. Use short cache-client timeouts, circuit breaking, and a capped bypass budget. If the source cannot safely accept the miss volume, it is better to serve a documented stale response or a temporary 503 than to make the database unavailable for every endpoint. The architecture decisions behind this are covered in the distributed cache system design guide.

Capacity Limits and Observability

Measure the origin budget before an incident. If a product database safely accepts 800 fallback reads per second and the application normally sends 200, only 600 reads per second remain for cache failures. Set a global refresh semaphore below that headroom, then set a smaller per-key limit of one or a few refreshers. The remainder must wait briefly, receive bounded-stale data, or be rejected. This is a product decision, not an implementation detail.

Alert on the shape of failure, not only on average cache hit ratio:

• Cache misses and origin queries grouped by key or key prefix reveal a hot-key stampede.
• Misses rising across many prefixes after a deploy or restart indicate an avalanche.
• High database not found reads, especially for random identifiers, indicate penetration.
• Refresh lease acquisition, lease contention, stale responses, negative-cache hits, and Bloom-filter rejects show whether protections are working.
• Origin p95 latency, database connection-pool utilization, queue depth, timeout rate, and retry rate reveal whether a cache miss is becoming a cascading failure.

Test the controls with load, not just a unit test. Expire one hot key during realistic concurrency, restart the cache in a staging environment, and send a stream of absent IDs. A successful test demonstrates that origin work stays bounded, stale responses never exceed their hard age, locks release after failure, and recovery does not require an operator to flush anything.

Interview Questions

1. What is the difference between a cache stampede and a cache avalanche?

A stampede is many requests rebuilding one expired hot key, while an avalanche is a broad wave of misses across many keys or an entire cache tier. I use request coalescing, stale-while-revalidate, and sometimes a per-key lease for the first. For the second, I add TTL jitter, controlled warming, load shedding, and an origin-capacity plan because locks alone cannot make the origin process many distinct keys. An avalanche can trigger many stampedes at the same time, which is why the distinction matters operationally.

2. Why is a Redis lock not enough to prevent a cache stampede?

A lock coordinates one cache key across application instances, but it does not validate input, protect a database from thousands of different misses, or decide what waiting requests receive. It can also fail if a holder pauses longer than its lease or releases a lease it no longer owns. I use a unique token and token-checked release, local coalescing to reduce lock traffic, bounded wait behavior, and an origin concurrency limit. For critical business invariants, I use a database transaction or fencing mechanism rather than treating a cache-refresh lease as a correctness guarantee.

3. When should a service serve stale cache data?

Only when the endpoint has an explicit freshness contract. Catalog copy, aggregate counters, and many feeds can often tolerate a short stale window, whereas account balances, inventory availability, authorization, and payment decisions may not. I define both a soft expiry, where stale data is allowed while refresh occurs, and a hard expiry, after which the response must reload or fail. That makes the availability-versus-freshness trade-off visible and testable.

4. How does negative caching stop cache penetration, and what is its risk?

It stores a short-lived sentinel for a key that was checked and found absent, so repeated requests do not repeat the origin lookup. The risk is hiding a newly created record until the sentinel expires, or accidentally sharing a permission-dependent absence between callers. I choose a TTL based on record creation latency, scope the cache key correctly, and validate malformed IDs before the cache. For a huge mostly-absent key space, I add a Bloom filter because it rejects definitely absent keys before the database.

5. How would you test cache-failure protection before production?

I would run three controlled experiments: expire one top key under peak-like concurrency, restart or switch key namespaces while traffic ramps gradually, and replay valid but absent IDs at a sustained rate. I would assert an upper bound on origin reads, database connections, request latency, stale-response age, and lock wait time. I would also kill a refresher after it acquires its lease to prove the lease expires safely and that no stale owner deletes a later owner’s lock. The point is to validate the recovery path, not merely observe that the cache returns values when healthy.

Conclusion

Cache failure handling is capacity engineering. A stampede needs one controlled refill for a hot key, an avalanche needs expiries and recovery work spread across time, and penetration needs impossible requests stopped before they reach the source of truth. Combine these controls with explicit stale-data contracts, bounded timeouts, and measured origin headroom. A cache is useful only when its failure mode is safer than its success mode is fast.

References

1. Redis cache-aside guidance
2. Redis distributed locks pattern
3. Redis Bloom filter documentation
4. MDN Cache-Control and stale-while-revalidate

YouTube Videos

1. Cache Stampede: Your Cache Makes Things Worse Under Load
2. Redis Cache Stampede Explained: Distributed Lock Solutions in .NET