
Webhook providers retry when an acknowledgement is slow or lost, so duplicate delivery is normal. The safe answer is an authenticated, durable, idempotent consumer—not “ignore duplicates in memory.” Connect this scenario to webhook delivery, message delivery guarantees, and API idempotency.
The Interview Question
“A provider delivers the same payment.succeeded webhook three times and occasionally out of order. How do you process it exactly once from the business perspective?”
Clarifying Questions
Ask whether events have immutable IDs and timestamps, whether signatures cover the raw body, expected volume, ordering guarantees, and the business state machine. Assume at-least-once delivery, 10,000 events per second, and no global ordering.
The Naive Design and Why It Fails
Checking a process-local set is lost on restart and differs across replicas. Acknowledge-before-persisting loses events when the worker crashes. Applying every event in arrival order can regress state when an older event arrives late.
Better Design
Verify the signature and timestamp, insert provider + event ID into a dedupe table with a unique key, and process the event in the same transaction as the business transition. Acknowledge after the durable commit; return success for an already-seen event.
flowchart TD
A[Webhook request] --> B[Verify signature and timestamp]
B --> C[Unique event ID insert]
C -->|duplicate| D[Return 2xx]
C -->|new| E[Apply state transition]
E --> F[Commit and enqueue outbox work]
F --> G[Return 2xx]
Ordering and Failure Handling
Store event version or sequence and reject transitions that move backward. If an event arrives before its predecessor, persist it as pending and reconcile from the provider. A dead-letter queue holds invalid or repeatedly failing events with the raw payload, error, and attempt metadata; replay uses the same event ID.
Inbox Schema and State Transitions
The dedupe record should describe processing, not merely prove that a request was seen:
CREATE TABLE webhook_inbox (
provider text NOT NULL,
event_id text NOT NULL,
resource_id text NOT NULL,
event_version bigint,
status text NOT NULL,
payload_hash text NOT NULL,
received_at timestamptz NOT NULL,
processed_at timestamptz,
error_code text,
PRIMARY KEY (provider, event_id)
);
Insert the inbox row and update the business aggregate in one local transaction when both use the same database. If the event is already PROCESSED, acknowledge it without repeating the side effect. If it is RETRYABLE, a worker may claim it with a lease; if it is FAILED, keep it visible for an operator or bounded replay. Store a hash so a malicious or misconfigured sender cannot reuse an event ID with a different payload.
For a resource-level state machine, compare the event version before applying it. An event for PAID at version 8 must not move an order back to AUTHORIZED at version 7. When a provider has no sequence number, use its documented creation time only as a hint and reconcile from the provider API; local arrival time is not an ordering guarantee.
Authentication, Replay, and Recovery
Verify the signature over the exact raw body before parsing or queueing the event. Enforce a timestamp tolerance, support secret rotation through a key ID, and keep a bounded replay cache for signed delivery IDs. Signature validation proves origin, not freshness: a captured valid request can still be replayed unless the event ID and timestamp are checked.
At 10,000 events per second, do not perform slow provider calls inside the HTTP handler. Persist the inbox row, enqueue an internal command, and return 2xx after the durable commit. A worker can then retry transient failures with backoff, preserve the original event ID, and send poison payloads to a DLQ. Replays should be selected by provider, resource, error class, and time range, with a dry-run count before side effects resume.
Measure duplicate rate, signature failures, inbox insert conflicts, processing age, pending predecessor age, DLQ depth, and reconciliation mismatches. Alert on the oldest unprocessed event and on a jump in duplicate delivery; a flat event count can hide a growing delay.
Scaling and Trade-offs
Partition consumers by account or resource ID when per-entity ordering matters. At 10,000 events/second, index the dedupe table by event ID and archive records after the provider retry window, retaining an audit reference. The database transaction adds latency but gives a clear business invariant; an asynchronous handler improves ingress availability but delays state visibility.
Interview Follow-Ups
1. Why acknowledge after processing?
Acknowledging first tells the provider the event is safe even if the process crashes before persistence. A later retry is preferable to silent loss. If processing is slow, persist an inbox record quickly, acknowledge, and process from the durable queue.
2. How do you verify signatures?
Use the raw request body, provider-specific signing secret, a constant-time comparison, and a timestamp tolerance to prevent replay. Rotate secrets with a key ID. Never trust an event merely because it came from an allowlisted IP.
3. What if two replicas process the same event?
The unique constraint lets one insert win; the other treats the conflict as a duplicate and returns success after confirming the winner’s state. Keep the business update in the same transaction as the inbox insert. This makes replicas converge without a distributed mutex.
4. How do you handle out-of-order events?
Use a versioned state machine and compare event sequence or creation time. Buffer events whose prerequisite is missing and reconcile after a bounded delay. Do not use wall-clock time alone when the provider offers a stronger sequence.
5. How do you replay safely?
Fix the consumer, select a bounded DLQ range, and replay with original event IDs. The dedupe record must distinguish an already-successful event from one that failed before completion. Measure replay rate and downstream side effects during the exercise.
Related Practice
Review duplicate payments, Kafka offsets, and observability.
References
YouTube Videos
- “Webhook Idempotency and Reliability” - Hussein Nasser [https://www.youtube.com/watch?v=DOFflggE_0Q]