Skip to content
ADevGuide Logo ADevGuide
Go back

Traffic Increases 100x: How Does the Architecture Evolve?

Updated:

By Pratik Bhuite | 10 min read

Hub: Java / Interview Fundamentals

Series: Backend Interview Mastery

Last updated: Sep 19, 2026

Part 66 of 88 in the Backend Interview Mastery

Key Takeaways

On this page
Reading Comfort:

Architecture evolving from a single service to a 100x scalable backend

“Traffic is 100x” is not a solution by itself. Clarify which requests, data, and SLO changed, then remove bottlenecks in order. Use scalability fundamentals, backpressure, and consistent hashing.

The Interview Question

“A product grows from 1,000 to 100,000 requests per second in six months. How does the architecture evolve without a risky rewrite?”

Clarifying Questions

Ask read/write mix, payload size, latency and availability targets, hot keys, geographic distribution, and growth shape. Assume 90% reads, 10% writes, 200 ms P99, 10 KB responses, and a bursty launch event.

The Naive Design and Why It Fails

Adding application replicas while every request hits one database primary simply moves the bottleneck. Adding a cache without invalidation creates stale business decisions. Sharding before measuring access patterns creates cross-shard joins and operational risk.

Better Evolution

Measure first, then add CDN/HTTP caching for public reads, a load balancer and stateless application replicas, Redis for hot computed data, read replicas, and asynchronous queues for noncritical writes. Partition data only when a single primary or tenant becomes the measured limit.

flowchart TD
    A[Clients] --> B[CDN and edge cache]
    B --> C[Load balancer]
    C --> D[Stateless app fleet]
    D --> E[Redis cache]
    D --> F[Read replicas]
    D --> G[Write primary]
    D --> H[Queue for async work]
    G --> I[Partitioned data when needed]

What Changes at Each Growth Stage

Treat the migration as a sequence of reversible constraints rather than a single rewrite.

  1. 1,000 to 5,000 RPS: remove obvious inefficiencies, add request metrics, bound the database pool, and make application instances stateless. This stage should be boring to operate and easy to roll back.
  2. 5,000 to 20,000 RPS: cache public reads at the edge, add cache-aside storage for hot objects, and move email, analytics, and indexing behind a queue. Keep writes on the authoritative primary until the access pattern is understood.
  3. 20,000 to 60,000 RPS: add read replicas, partition the cache by key space, and isolate expensive tenants or endpoints with bulkheads. Replication lag becomes a product concern, so APIs should declare whether stale reads are acceptable.
  4. 60,000 to 100,000 RPS: introduce regional routing, partition only the tables that have a measured hot spot, and enforce admission control before requests consume database connections. At this point, capacity limits and overload responses are part of the API contract.

At every stage, keep a compatibility layer around old and new paths. Dual-read or shadow traffic can compare results before switching authority; dual-write should be used only with a reconciliation plan because it creates two sources of truth.

Finding the Real Bottleneck

Start with four signals for each tier: traffic, latency percentiles, errors, and saturation. A database CPU graph alone cannot tell you whether the cause is a bad query, a connection storm, a lock, or a traffic mix change. Break down requests by route, tenant, cache status, payload size, and region, then compare the slowest cohort with the baseline.

The capacity question is also about bytes. At 100,000 RPS and 10 KB responses, the origin would emit roughly 1 GB/s before compression. A CDN can remove most public response bytes from the application fleet, while request bodies, authenticated reads, and writes still reach the origin. This distinction prevents an edge-cache success from hiding a write-path bottleneck.

Capacity Math and Rollout

At 100,000 RPS with 90% cacheable reads, a 90% hit ratio removes 81,000 database reads per second. If one app instance safely handles 500 RPS, start with 240 instances for headroom rather than 200. Load-test each stage, canary changes, and keep a rollback path.

Failure Handling and Trade-offs

Rate-limit abusive clients, shed optional work, and protect the database with bounded pools. Eventual consistency and asynchronous writes improve throughput but alter user-visible timing. Global routing improves latency but introduces replication and failover complexity.

During a launch, define overload behavior before the launch happens. Return 429 when a caller exceeds a known quota, 503 with a retry hint when the service is temporarily full, and preserve a small operator or checkout budget so one noisy tenant cannot consume every connection. Disable recommendations, exports, and expensive fan-out first; do not silently drop a durable order or payment command.

For a regional failure, route only traffic whose data contract permits it. A read-only catalog can fail over to a replicated projection, while a reservation or payment write may need one authoritative region. Record the routing decision and recovery point so the team can reconcile writes after the partition instead of assuming active-active correctness.

Interview Follow-Ups

1. What do you scale first?

Identify the saturated resource with rate, latency, errors, and saturation metrics. If the database is hot on reads, cache or add replicas; if CPU is hot in the app, optimize or scale stateless workers. Capacity math prevents an irrelevant change.

2. When does caching stop helping?

When the working set is too large, keys are uncacheable, invalidations are frequent, or hot-key skew overloads one node. Measure hit ratio, origin load, key distribution, and stale-read impact. Add request coalescing or partition hot keys when appropriate.

3. Why introduce a queue?

It absorbs bursts and lets workers process noncritical work at a controlled rate. It adds lag, at-least-once delivery, and operational state. Use it only when the business can tolerate asynchronous completion and provide status visibility.

4. How do you shard safely?

Choose a stable partition key, route consistently, migrate gradually, and design for cross-partition queries. Keep a ring or directory versioned and observable. Sharding is a data-model change, not merely adding servers.

5. How do you test 100x growth?

Replay realistic traffic with hot-key and burst distributions, then inject dependency failures. Measure P99, queue age, database CPU, replication lag, and recovery time. Test rollback and capacity limits before launch.

Continue with cache failure handling, database CPU saturation, and the real-time analytics design.

References

  1. Google SRE: Handling overload
  2. AWS: Reliability Pillar

YouTube Videos

  1. “How to Scale a System 100x” - Gaurav Sen [https://www.youtube.com/watch?v=Ibf7wKf8MqU]

Share this post on:

Next in Series

Continue through the [object Object] with the next recommended article.

Related Posts

Keep Learning with New Posts

Subscribe through RSS and follow the project to get new series updates.

Was this guide helpful?

Share detailed feedback

Previous Post
Redis Goes Down: Should the Application Fail?
Next Post
Functional vs Non-Functional Requirements in System Design Interviews