Skip to content
ADevGuide Logo ADevGuide
Go back

System Design Interview: Logging System

By Pratik Bhuite | 16 min read

Hub: System Design / Interview Questions

Series: System Design Interview Series

Last verified: Sep 19, 2026

Part 9 of 24 in the System Design Interview Series

Key Takeaways

On this page
Reading Comfort:

System Design Interview: Logging System

Logging System is a classic system design interview prompt because it forces you to make a clear choice about absorbing bursty high-volume log ingestion. Use the system design interview cheatsheet for the reusable framework, then compare this design with the System Design hub and Backend Interview Mastery.

Table of Contents

Open Table of Contents

Interview Framework: How to Approach This Problem

Allocate five minutes to clarify the user promise, two minutes to state assumptions, ten minutes to draw the architecture, twenty minutes to deep-dive into absorbing bursty high-volume log ingestion, and the final ten minutes to scale, secure, and test it. Think aloud, name uncertainty, and ask the interviewer whether the trade-off matches the product requirement.

Step 1: Clarifying Requirements

Ask: What is the core user action? Is completion synchronous or eventual? What correctness guarantee matters most? What is the peak rate and payload size? Which regions and tenants are in scope? What is the acceptable P99 and availability target? What must be auditable?

For this design, support the primary logging system flow, retries, status inspection, and operator recovery. Keep optional ranking, multi-region writes, and advanced analytics out of the MVP unless the interviewer asks for them.

Step 2: Core Assumptions and Constraints

Assume 5 million events/sec; 30-day hot search; multi-terabyte daily volume. Choose an availability target of 99.9% for the control plane and a logging system-appropriate user SLO. Requests carry an authenticated tenant or user identity, and every mutating request has an idempotency key. Eventual consistency is acceptable for status views, but the authoritative invariant is checked transactionally.

Step 3: High-Level Architecture

The main components are Agents, Ingest Gateway, Buffer, Stream Enricher, Hot Index, Object Archive, Query API. Keep the API stateless, isolate asynchronous work behind durable queues, and make the system of record explicit.

System Flow Diagram

flowchart TD
    A[Agents and SDKs] --> B[Regional Collectors]
    B --> C[Durable Log Buffer]
    C --> D[Partitioned Stream]
    D --> E[Parser and Enricher]
    E --> F[Hot Search Index]
    E --> G[Cold Object Storage]
    F --> H[Query and Alert API]
    B -. overload .-> I[Sampling and Shedding]

Architecture Walkthrough

  1. Agents batch and compress logs locally, attaching tenant, service, trace, and schema metadata.
  2. Regional collectors authenticate producers and write to a durable buffer before acknowledging accepted data.
  3. Stream workers parse, redact secrets, normalize timestamps, and route records to hot indexes or cold storage by retention policy.
  4. Query APIs search only the hot tier by default; a bounded cold query is explicit and cost-controlled.
  5. Sampling and load shedding protect the platform during bursts while preserving error and audit streams.
  6. Retention jobs delete or compact data by tenant policy, and a dead-letter stream preserves malformed records for repair.

The ingestion acknowledgment is the key contract: it means the platform durably accepted the event, not that it is already searchable. That distinction keeps burst handling and user expectations honest.

Step 4: The Hardest Problem - absorbing bursty high-volume log ingestion

The difficult case is a slow or duplicated operation that overlaps a failure. Three approaches are plausible:

Approach 1: Synchronous request chain

The API calls every dependency before responding. It is easy to explain and provides immediate feedback, but one slow dependency consumes request capacity and a lost response makes the outcome ambiguous. It fails under bursty load.

Approach 2: Best-effort asynchronous events

The API emits an event and returns immediately. This absorbs spikes, but without a durable state machine, duplicate delivery and lost events leave the user unable to know whether work completed. It is not enough for the authoritative invariant.

Approach 3: Durable state machine with leased workers

Persist the intent, publish through an outbox, claim work with a lease, and make each transition idempotent. A worker can resume after a crash, while the status API exposes PENDING, SUCCEEDED, and FAILED states. This is the chosen approach because it makes failure visible and replayable.

Step 5: append-only ingestion with hot and cold storage tiers

Choose append-only ingestion with hot and cold storage tiers. The decision trades a small amount of workflow complexity and eventual completion for bounded API latency, independent scaling, and a recoverable audit trail. Use a monotonically increasing version on each state transition so stale workers cannot overwrite a newer result.

A minimal worker contract is:

async function process(command: Command) {
  const claimed = await store.claim(
    command.id,
    command.leaseId,
    command.version
  );
  if (!claimed) return; // another worker won, or the command is already complete
  try {
    const result = await performIdempotently(command);
    await store.complete(command.id, result, command.version + 1);
  } catch (error) {
    await store.failOrRetry(command.id, classify(error), command.version + 1);
  }
}

Step 6: Database Design and Storage

Classify data by access temperature. Hot state and leases need a low-latency transactional store; recent results and indexes are warm; raw payloads, audit history, and old artifacts belong in object or columnar storage.

Use these core records: log_records, streams, tenants, retention_policies. Include an immutable ID, tenant or owner, current status, version, idempotency key, created and updated timestamps, correlation ID, and error code. Index the idempotency key uniquely, status plus next-attempt time for workers, and owner plus created time for queries. Never use a cache as the sole source of the correctness invariant.

Step 7: Scaling the System

Start with 5 million events/sec; 30-day hot search; multi-terabyte daily volume. Identify four bottlenecks:

  1. Ingress: use stateless replicas, connection limits, and rate limits.
  2. State store: partition by tenant or entity ID, batch noncritical writes, and keep transactions short.
  3. Queue/workers: partition by ordering key, autoscale on queue age, and apply backpressure when downstream capacity is full.
  4. Read path: maintain a projection, cache safe immutable results, and paginate history.

If one worker handles 100 operations/second and peak demand is 10,000 operations/second, 100 workers meet the raw rate; provision 2x for failures and burst headroom. If each operation fans out to three calls, budget 30,000 downstream calls/second before retries. State the math and then ask which number the interviewer wants to optimize.

Step 8: Security and Permissions

Authenticate users or service principals at the edge and authorize every state transition against tenant ownership. Scope idempotency keys to the tenant, encrypt sensitive payloads at rest, and redact secrets from logs. Sign internal events, rotate keys, and use short-lived worker credentials.

For operator replay, require a separate role, a reason, an approval or ticket, and an immutable audit record. Rate-limit status and mutation endpoints, validate uploaded or embedded content, and prevent a tenant from selecting another tenant’s IDs.

Step 9: Handling Edge Cases

  1. Client timeout: return the stored status for the same idempotency key; never blindly repeat the side effect.
  2. Worker crash: lease expiry makes the command claimable; version checks prevent stale completion.
  3. Duplicate event: the inbox or unique command key turns it into a no-op with the original result.
  4. Downstream timeout: honor the remaining deadline, retry only idempotent transient failures, and move exhausted work to a DLQ.
  5. Region outage: fail over reads and accept writes only where the consistency contract remains true; reconcile after recovery.

Step 10: Performance Optimizations

Use Kafka, object storage, columnar indexes where it matches the workload. Batch queue acknowledgements, compress large payloads, cache immutable metadata, and keep hot worker state in memory with durable checkpoints. Use adaptive concurrency so workers do not overwhelm a dependency.

Measure end-to-end latency, queue age, success rate, retry amplification, state-store CPU, and stale projection age. A faster average that increases duplicate work or tail latency is not an optimization.

Real-World Implementations

Elastic and Splunk demonstrate the same principles at production scale: durable buffering, partitioned ingestion, hot and cold retention, and explicit recovery. A practical stack can combine a log buffer, object storage, columnar indexes, redaction, and OpenTelemetry correlation. The exact vendor is less important than the durable-acceptance contract, failure boundary, and operational evidence.

Common Interview Follow-Up Questions

1. How do you guarantee idempotency?

Persist a scoped idempotency key with a unique constraint and store the first result. Every retry returns that result or a clearly pending state. Downstream calls receive the same key so provider retries are safe too.

2. What happens when the queue is unavailable?

Commit the authoritative intent only if the outbox can record the event in the same transaction. A relay can publish later. If the state store is unavailable, reject new work rather than acknowledging an intent that cannot be recovered.

3. How do you handle ordering?

Partition commands by the entity whose order matters and attach a version to every transition. Workers reject a version older than the stored one. If global ordering is requested, explain its throughput and availability cost.

4. How do you scale across regions?

Prefer local reads and regionally partitioned writes when the business allows it. Use a clear home region or consensus authority for single-writer invariants, and replicate projections asynchronously. During a partition, state which operations fail and how reconciliation works.

5. How do you operate the system?

Define SLOs for API latency, completion time, freshness, and recovery. Alert on oldest queue age, stuck leases, retry amplification, and invariant violations. Provide dashboards, runbooks, bounded replay, and a kill switch for expensive work.

Conclusion

A strong logging system interview answer is a sequence of explicit choices: requirements, one authoritative invariant, a durable state machine, idempotent commands, bounded asynchronous work, and measurable recovery. Draw the happy path, then spend most of the interview on the failure that makes the design non-obvious.

References

  1. AWS architecture guidance
  2. OpenTelemetry documentation
  3. Google SRE: Handling overload

YouTube Videos

  1. “System Design Interview: Logging System” - Hello Interview [https://www.youtube.com/watch?v=Qg4aPDZFbkA]

Share this post on:

Next in Series

Continue through the [object Object] with the next recommended article.

Related Posts

Keep Learning with New Posts

Subscribe through RSS and follow the project to get new series updates.

Was this guide helpful?

Share detailed feedback

Previous Post
System Design Interview: Large-Scale Inventory System
Next Post
System Design Interview: Metrics Platform