
A producer emits 20,000 events per second while the database-backed consumer can process 5,000. If the system accepts everything into unbounded memory, it eventually crashes; if it keeps retrying, it amplifies load. Backpressure makes the capacity boundary explicit and tells producers when to slow, buffer, reject, or shed work.
This guide explains bounded queues, flow control, rate limits, load shedding, Kafka lag, and recovery. Continue from Kafka consumer lag, retries with jitter, and caching stampede protection.
Table of Contents
Open Table of Contents
What Backpressure Protects
Backpressure protects finite resources: memory, thread pools, sockets, database connections, broker partitions, and downstream rate limits. It is a feedback loop, not merely a queue. The consumer or resource owner exposes capacity, and the producer adjusts work before the buffer becomes an outage.
flowchart TD
A[Producer] --> B{Capacity available?}
B -->|Yes| C[Accept or process]
B -->|Limited| D[Bounded buffer or queue]
D --> E{Buffer threshold}
E -->|Below high-water mark| A
E -->|Above high-water mark| F[Slow producer]
E -->|Full| G[Reject, shed, or drop policy]
C --> H[Consumer]
H --> I[Downstream resource]
I -->|Saturation signal| B
Without a policy, memory becomes an accidental buffer and garbage collection becomes the first visible symptom. A bounded queue may increase latency, but it gives operators a finite recovery problem instead of an unbounded crash.
Backpressure Strategies
| Strategy | Best for | Trade-off |
|---|---|---|
| Block or await | In-process pipelines | Preserves work but consumes caller capacity |
| Bounded queue | Burst smoothing | Oldest work waits and may expire |
| Rate limit | Public APIs and dependencies | Rejects excess demand explicitly |
| Load shedding | Optional or stale work | Improves core availability by dropping work |
| Sample or coalesce | Metrics and duplicate updates | Loses intermediate events |
| Pause consumer/producer | Brokers and streams | Preserves data but increases lag |
Choose the loss policy from business value. Dropping a duplicate cache refresh may be safe; dropping a payment command is not. Queue entries need a maximum age and a dead-letter or reconciliation path when they expire.
Queue and Stream Design
Kafka exposes backpressure indirectly through consumer lag, fetch limits, pause/resume, and producer quotas. A consumer that processes slower than ingress should scale workers only when partitions and downstream capacity allow it; otherwise more consumers move pressure into the database. RabbitMQ uses prefetch and acknowledgements to limit in-flight deliveries.
Set high- and low-water marks with hysteresis. If producers resume exactly at the same threshold at which they paused, the system can oscillate. Alert on age and drain rate, not only queue count. A queue with 100 items may be healthy or hours behind depending on ingress and work cost.
API and Worker Examples
For a synchronous API, return 429 Too Many Requests with a retry hint when admission control rejects work. For a background endpoint, persist a job only when capacity or retention is available and return a job ID. Never accept work into memory and promise that a later worker will find it.
async function enqueueJob(job: Job) {
const depth = await queue.depth();
if (depth >= HIGH_WATER_MARK) {
metrics.increment("jobs.rejected_backpressure");
throw new TooManyRequestsError({ retryAfterSeconds: 10 });
}
// Persist before responding so accepted work survives a process crash.
return queue.publish(job);
}
Workers should limit concurrency to the downstream’s safe capacity and release a slot only after the side effect completes. A semaphore, connection pool, or consumer prefetch setting can be the local backpressure boundary; document which one owns the limit.
Recovery and Observability
Track queue depth, oldest age, ingress and consume rates, rejection rate, active work, downstream saturation, memory, and time to drain. During recovery, increase capacity gradually and verify that the dependency is not already saturated. A backlog drain rate of 2,000 jobs per second against a 100,000-job backlog is roughly 50 seconds only if new ingress is zero and the rate remains stable.
Test a full queue, slow consumer, consumer crash, producer burst, dependency rate limit, expired work, and recovery oscillation. Include a manual kill switch for non-critical producers; graceful degradation is easier when operators can stop optional work before core traffic is affected.
Interview Questions
1. What is backpressure?
Backpressure is feedback that limits or reshapes producer work when a consumer or downstream resource cannot keep up. It can block, buffer, slow, reject, shed, or coalesce work. The goal is to preserve system stability at a known capacity boundary.
2. Why are unbounded queues dangerous?
They convert sustained overload into unbounded memory, latency, and eventually process failure. An unbounded queue also hides the time at which the system stopped meeting its business SLO. Bounded queues force an explicit admission or loss decision.
3. Should you add more consumers when lag grows?
Only if partitions, CPU, and downstream capacity are available. More consumers cannot exceed partition parallelism and can overload the database or API they call. Check per-partition lag and the bottleneck before scaling the consumer group.
4. When is load shedding acceptable?
When work is optional, stale, duplicate, or lower priority than the core user journey. Make the policy explicit and observable, and never silently drop durable business commands. Critical work should use persistence, retry, reconciliation, or a dead-letter path instead.
5. How do you prevent oscillation around a queue threshold?
Use separate high- and low-water marks, a minimum pause or resume interval, and jittered producer recovery. This hysteresis prevents every producer from turning on at the same instant the queue becomes slightly healthy. Monitor state transitions and queue age to tune the control loop.
Conclusion
Backpressure makes overload a controlled state instead of an accidental crash. Bound queues, expose capacity, select a business-aware loss policy, limit worker concurrency, and measure age and drain rate. The best design lets core work remain correct while optional work slows or stops during a dependency failure.
References
- Reactive Manifesto: Message-driven and resilient systems
- Kubernetes: Resource management for Pods and Containers
- Apache Kafka: Consumer configs
YouTube Videos
- “Backpressure in Reactive Systems” - Reactive Programming [https://www.youtube.com/watch?v=8quZUZWm4oM]