Skip to content
ADevGuide Logo ADevGuide
Go back

Backend Interview Scenario: High P99 Latency

By Pratik Bhuite | 11 min read

Hub: Java / Interview Fundamentals

Series: Backend Interview Mastery

Last verified: Sep 19, 2026

Part 78 of 88 in the Backend Interview Mastery

Key Takeaways

On this page
Reading Comfort:

Backend interview scenario: High P99 Latency

This scenario asks you to diagnose high P99 latency by separating queueing, CPU, database, cache, and downstream time budgets. Treat it as an incident and design prompt: state the invariant, expose the failure boundary, and choose a mitigation that can be rolled back. Continue from the Backend Interview Mastery series and the system design interview cheatsheet.

Table of Contents

Open Table of Contents

1. Interview Question

“An on-call engineer reports that we must diagnose high P99 latency by separating queueing, CPU, database, cache, and downstream time budgets. How would you make the system correct under concurrency, retries, and partial failure?”

2. Clarifying Questions

Ask what the user-visible invariant is, what is authoritative, the peak rate, acceptable latency, retry policy, ordering requirement, failure scope, and recovery objective. Assume Metrics, tracing, profiler, pool dashboards, dependency budgets.

3. Naive Design

Start with one synchronous service and a direct call to the dependency. It is easy to deploy, but it couples request latency to every downstream operation and leaves no durable record for work that timed out. A process-local cache or lock also fails when requests land on another replica.

4. Why It Fails

The naive path violates correctness when two requests observe the same state, a response is lost after a commit, or a worker dies mid-operation. It also scales the wrong resource: retries and unbounded concurrency amplify load exactly when the dependency is unhealthy.

5. Better Design

Persist an intent or event before performing the side effect, attach an idempotency key, and make state transitions monotonic. Use a bounded queue for asynchronous work, a lease for worker ownership, and a dead-letter path for exhausted attempts. Return a pending status when the outcome is uncertain instead of guessing.

6. Architecture

The core components are P50 50 ms; P99 2 seconds; 10,000 RPS. Keep the request path thin and route work through durable state and a queue.

flowchart TD
    A[Request] --> B[Gateway Deadline]
    B --> C[Trace Context]
    C --> D[App Handler]
    D --> E[Connection Pool]
    D --> F[Cache]
    D --> G[Database or Downstream]
    G -. slow .-> H[Timeout and Fallback]
    H --> I[Latency SLO Dashboard]
  1. Set a deadline at the gateway and propagate the remaining budget.
  2. Use traces and pool metrics to separate queue, CPU, database, and downstream time.
  3. Apply the smallest reversible mitigation at the measured bottleneck.
  4. Shed optional work and serve a bounded fallback when the deadline is exhausted.
  5. Compare tail latency and saturation before and after the change.

7. Data and Race Analysis

Use a state table with id, owner or tenant, idempotency key, status, version, lease expiry, attempt count, and timestamps. Add a unique index for the key and a compare-and-set on version. If two workers race, only the worker holding the current lease and version may commit. Keep the system of record authoritative; caches and projections may be stale.

8. Failure Handling

A timeout becomes UNKNOWN or PENDING until the system can query or reconcile the dependency. Retry only transient, idempotent operations with exponential backoff and jitter. Expired leases make work claimable, while a DLQ preserves payload, error, correlation ID, and attempt history. Recovery starts with the oldest pending item, not the newest alert.

9. Scaling and Capacity

Assume 10,000 requests/second with a 50 ms P50 and 2 second P99; budget queue, pool, database, and downstream time separately. A 100 ms downstream regression can consume the entire tail budget when fan-out is serial. Scale only the measured bottleneck.

10. Trade-offs

Synchronous completion gives immediate feedback but couples availability and latency. Asynchronous completion absorbs bursts and enables replay but requires a status API and eventual-consistency language. Strong serialization protects the invariant but can reduce throughput; partitioning improves scale while narrowing the ordering guarantee.

11. Observability

Track request rate, error rate, P50/P95/P99, queue age, oldest pending work, lease expiry, retry amplification, state-store saturation, and invariant violations. Correlate logs, traces, and events with one ID. Alert on trend and age, not only instantaneous error percentage.

12. Interview Follow-Ups

Scenario-Specific Follow-Ups

1. How do you decompose a P99 spike?

Break the trace into queue wait, CPU, connection-pool wait, database time, downstream time, and serialization. Compare the tail with a baseline; an average latency chart can hide the saturated component.

2. When should you add capacity?

Add capacity only after identifying the bottleneck. More workers do not fix a serialized database lock or a slow dependency, and can increase queueing and retry load.

3. How do you prove the fix worked?

Run the same representative load, compare P50/P95/P99 and error rate, and watch saturation after the change. Keep a rollback trigger tied to the original SLO, not just a lower average.

1. How do you make retries safe?

Store a scoped idempotency key and the first result under a unique constraint. Pass that key to downstream providers and return the stored result on duplicate requests. A retry after an uncertain timeout therefore converges on one business outcome.

2. What if the queue is unavailable?

Commit the intent only when an outbox row is written in the same transaction. A relay can publish later. If the state store itself is unavailable, reject new work rather than acknowledging an unrecoverable request.

3. How do you handle a hot key?

Coalesce concurrent requests, shard or replicate the hot key when semantics allow, and rate-limit the caller. Do not simply increase concurrency against one serialized row. Measure per-key skew separately from fleet averages.

4. How do you recover after a region or dependency outage?

Pause nonessential work, cap retries, fail over only where the data contract permits, and reconcile after recovery. Replay a bounded range with the original IDs. Document which user operations return pending or retryable states.

5. What would you test before launch?

Run concurrent invariant tests, crash workers after each state transition, inject timeouts and duplicate events, and load-test the hottest partition. Verify dashboards, rollback, DLQ replay, and operator permissions. Test recovery time as a first-class SLO.

Read backpressure, timeouts, and observability.

References

  1. Google SRE: Handling overload
  2. AWS: Timeouts, retries and backoff
  3. OpenTelemetry documentation

YouTube Videos

  1. “Backend Distributed Systems Interview Scenario” - Hello Interview [https://www.youtube.com/watch?v=Ibf7wKf8MqU]

Share this post on:

Next in Series

Continue through the [object Object] with the next recommended article.

Related Posts

Keep Learning with New Posts

Subscribe through RSS and follow the project to get new series updates.

Was this guide helpful?

Share detailed feedback

Previous Post
Backend Interview Scenario: Flash-Sale Inventory Protection
Next Post
Backend Interview Scenario: Message-Processing Failure and DLQ Replay