
“Traffic is 100x” is not a solution by itself. Clarify which requests, data, and SLO changed, then remove bottlenecks in order. Use scalability fundamentals, backpressure, and consistent hashing.
The Interview Question
“A product grows from 1,000 to 100,000 requests per second in six months. How does the architecture evolve without a risky rewrite?”
Clarifying Questions
Ask read/write mix, payload size, latency and availability targets, hot keys, geographic distribution, and growth shape. Assume 90% reads, 10% writes, 200 ms P99, 10 KB responses, and a bursty launch event.
The Naive Design and Why It Fails
Adding application replicas while every request hits one database primary simply moves the bottleneck. Adding a cache without invalidation creates stale business decisions. Sharding before measuring access patterns creates cross-shard joins and operational risk.
Better Evolution
Measure first, then add CDN/HTTP caching for public reads, a load balancer and stateless application replicas, Redis for hot computed data, read replicas, and asynchronous queues for noncritical writes. Partition data only when a single primary or tenant becomes the measured limit.
flowchart TD
A[Clients] --> B[CDN and edge cache]
B --> C[Load balancer]
C --> D[Stateless app fleet]
D --> E[Redis cache]
D --> F[Read replicas]
D --> G[Write primary]
D --> H[Queue for async work]
G --> I[Partitioned data when needed]
What Changes at Each Growth Stage
Treat the migration as a sequence of reversible constraints rather than a single rewrite.
- 1,000 to 5,000 RPS: remove obvious inefficiencies, add request metrics, bound the database pool, and make application instances stateless. This stage should be boring to operate and easy to roll back.
- 5,000 to 20,000 RPS: cache public reads at the edge, add cache-aside storage for hot objects, and move email, analytics, and indexing behind a queue. Keep writes on the authoritative primary until the access pattern is understood.
- 20,000 to 60,000 RPS: add read replicas, partition the cache by key space, and isolate expensive tenants or endpoints with bulkheads. Replication lag becomes a product concern, so APIs should declare whether stale reads are acceptable.
- 60,000 to 100,000 RPS: introduce regional routing, partition only the tables that have a measured hot spot, and enforce admission control before requests consume database connections. At this point, capacity limits and overload responses are part of the API contract.
At every stage, keep a compatibility layer around old and new paths. Dual-read or shadow traffic can compare results before switching authority; dual-write should be used only with a reconciliation plan because it creates two sources of truth.
Finding the Real Bottleneck
Start with four signals for each tier: traffic, latency percentiles, errors, and saturation. A database CPU graph alone cannot tell you whether the cause is a bad query, a connection storm, a lock, or a traffic mix change. Break down requests by route, tenant, cache status, payload size, and region, then compare the slowest cohort with the baseline.
The capacity question is also about bytes. At 100,000 RPS and 10 KB responses, the origin would emit roughly 1 GB/s before compression. A CDN can remove most public response bytes from the application fleet, while request bodies, authenticated reads, and writes still reach the origin. This distinction prevents an edge-cache success from hiding a write-path bottleneck.
Capacity Math and Rollout
At 100,000 RPS with 90% cacheable reads, a 90% hit ratio removes 81,000 database reads per second. If one app instance safely handles 500 RPS, start with 240 instances for headroom rather than 200. Load-test each stage, canary changes, and keep a rollback path.
Failure Handling and Trade-offs
Rate-limit abusive clients, shed optional work, and protect the database with bounded pools. Eventual consistency and asynchronous writes improve throughput but alter user-visible timing. Global routing improves latency but introduces replication and failover complexity.
During a launch, define overload behavior before the launch happens. Return 429 when a caller exceeds a known quota, 503 with a retry hint when the service is temporarily full, and preserve a small operator or checkout budget so one noisy tenant cannot consume every connection. Disable recommendations, exports, and expensive fan-out first; do not silently drop a durable order or payment command.
For a regional failure, route only traffic whose data contract permits it. A read-only catalog can fail over to a replicated projection, while a reservation or payment write may need one authoritative region. Record the routing decision and recovery point so the team can reconcile writes after the partition instead of assuming active-active correctness.
Interview Follow-Ups
1. What do you scale first?
Identify the saturated resource with rate, latency, errors, and saturation metrics. If the database is hot on reads, cache or add replicas; if CPU is hot in the app, optimize or scale stateless workers. Capacity math prevents an irrelevant change.
2. When does caching stop helping?
When the working set is too large, keys are uncacheable, invalidations are frequent, or hot-key skew overloads one node. Measure hit ratio, origin load, key distribution, and stale-read impact. Add request coalescing or partition hot keys when appropriate.
3. Why introduce a queue?
It absorbs bursts and lets workers process noncritical work at a controlled rate. It adds lag, at-least-once delivery, and operational state. Use it only when the business can tolerate asynchronous completion and provide status visibility.
4. How do you shard safely?
Choose a stable partition key, route consistently, migrate gradually, and design for cross-partition queries. Keep a ring or directory versioned and observable. Sharding is a data-model change, not merely adding servers.
5. How do you test 100x growth?
Replay realistic traffic with hot-key and burst distributions, then inject dependency failures. Measure P99, queue age, database CPU, replication lag, and recovery time. Test rollback and capacity limits before launch.
Related Practice
Continue with cache failure handling, database CPU saturation, and the real-time analytics design.
References
YouTube Videos
- “How to Scale a System 100x” - Gaurav Sen [https://www.youtube.com/watch?v=Ibf7wKf8MqU]