
One slow dependency can consume every request thread in its callers and create a cascading outage. Demonstrate a deadline budget, bounded concurrency, a circuit breaker, and a user-appropriate fallback. See timeouts, retries, and circuit breakers.
The Interview Question
“The pricing service starts timing out. Your checkout API has a five-second SLO. How do you contain the failure and decide what customers see?”
Clarifying Questions
Ask whether pricing is required to authorize checkout, the dependency’s error budget, request fan-out, and whether stale prices are allowed. Assume a 2-second caller deadline, three downstream calls, and no safe stale price for payment capture.
The Naive Design and Why It Fails
Retrying each call three times with a five-second timeout multiplies load and guarantees the caller misses its SLO. Unbounded connection pools and synchronous waits spread the outage to unrelated routes. Returning cached prices without a freshness contract can charge the wrong amount.
Better Design
Propagate a deadline, allocate a per-call budget, limit concurrent calls, and open a circuit after a rolling failure threshold. Fail checkout closed when price correctness is required, but allow catalog browsing to show a clearly labeled stale value.
flowchart TD
A[Request deadline 2s] --> B[Bulkhead for pricing]
B --> C{Circuit state}
C -->|Closed| D[One bounded call]
C -->|Open| E[Fast fallback or retryable error]
D --> F{Response before deadline?}
F -->|Yes| G[Continue checkout]
F -->|No| E
Budget the Request, Not Just the Socket
The five-second endpoint SLO is not a five-second timeout for every dependency. In this checkout path, choose the stated two-second caller deadline so the client does not wait through the entire SLO. If local validation and persistence need 300 ms, leave roughly 1.7 seconds for the pricing call and reserve the remaining time for response serialization and network variance. A fan-out of three pricing calls should either run in parallel under one shared deadline or be reduced to one batch request; giving each child its own independent five-second timeout defeats the caller’s budget.
Propagate an absolute deadline, not only a timeout duration. Each service computes the remaining time and refuses work when the budget is already exhausted:
deadline = request.deadline
remaining = deadline - now()
if remaining <= safety_margin:
return retryable_timeout
call pricing with min(remaining, pricing_budget)
retry only if the operation is idempotent and remaining time permits backoff
The same boundary applies to connection acquisition, DNS, TLS, database queries, and response writes. A socket timeout without a pool timeout still allows requests to pile up waiting for a connection.
Choosing the Fallback by Business Invariant
| Caller operation | Safe fallback | Unsafe fallback |
|---|---|---|
| Product browsing | Versioned stale price with a visible freshness timestamp | Presenting a stale value as the checkout total |
| Cart display | Show the last known estimate and request revalidation | Capturing payment against an unverified total |
| Checkout | Retryable error or PENDING_REPRICE state | Guessing a price to keep the UI successful |
| Recommendations | Omit the module or show a cached list | Blocking the whole page until recommendations recover |
This is why “always return cached data” is not resilience. The fallback must preserve the invariant that matters to that operation.
Containing Recovery Traffic
When the dependency recovers, do not close the circuit and release every waiting request. Keep a small half-open probe set, increase it gradually, and enforce a retry budget at the caller. A caller making 1,000 original requests per second with three retries can create 4,000 QPS during recovery. Track original and retry traffic separately, and coordinate the reopening rate with the downstream service’s advertised capacity.
Use bulkheads per route or dependency, not one process-wide pool. A slow pricing call should not consume the connections used for health checks, order reads, or operator traffic. Every rejected request should carry a correlation ID and a retryable error that clients can handle without an immediate tight loop.
Failure Handling and Recovery
Use jittered, idempotent retries only for transient errors and only within the remaining deadline. Emit dependency metrics and trip the circuit before all threads block. Half-open probes should be few; close the circuit only after success and latency recover.
Scaling and Trade-offs
If 1,000 requests per second each fan out to three pricing calls, the dependency sees 3,000 QPS before retries. A retry budget of 10% caps amplification at 300 extra QPS. Bulkheads protect other endpoints, but rejected work reduces feature availability; explain that trade-off.
Interview Follow-Ups
1. How do you choose a timeout?
Start from the endpoint SLO, subtract local processing and downstream budgets, and use measured tail latency. Propagate the deadline so nested services cannot outlive the request. Revisit it as traffic and latency distributions change.
2. When should you retry?
Retry only transient failures, with bounded exponential backoff and jitter, when the operation is idempotent or has a deduplication key. Do not retry validation errors, authentication failures, or an expired deadline.
3. What is a bulkhead?
It limits resources for one dependency or route so its failure cannot consume the entire process. Use separate pools, queues, or concurrency semaphores. Tie the limit to capacity and observed rejection rates.
4. What fallback is safe for payments?
If the amount depends on the dependency, a stale or guessed value is unsafe. Return a retryable error or place the order in a pending state that re-prices before capture. A fallback is safe only when its correctness contract is explicit.
5. How do you avoid a retry storm during recovery?
Enforce caller retry budgets, add jitter, use circuit half-open limits, and gradually increase traffic. Coordinate with dependency rate limits. Monitor retries separately from original calls so amplification is visible.
Related Practice
Practice Redis failure handling, backpressure, and service discovery.
References
YouTube Videos
- “Circuit Breaker Pattern Explained” - Hussein Nasser [https://www.youtube.com/watch?v=rvKKj7JU92g]