Skip to content
ADevGuide Logo ADevGuide
Go back

Duplicate Webhooks: How Do You Process Them Safely?

Updated:

By Pratik Bhuite | 10 min read

Hub: Java / Interview Fundamentals

Series: Backend Interview Mastery

Last updated: Sep 19, 2026

Part 67 of 88 in the Backend Interview Mastery

Key Takeaways

On this page
Reading Comfort:

An idempotent webhook consumer deduplicating retried events

Webhook providers retry when an acknowledgement is slow or lost, so duplicate delivery is normal. The safe answer is an authenticated, durable, idempotent consumer—not “ignore duplicates in memory.” Connect this scenario to webhook delivery, message delivery guarantees, and API idempotency.

The Interview Question

“A provider delivers the same payment.succeeded webhook three times and occasionally out of order. How do you process it exactly once from the business perspective?”

Clarifying Questions

Ask whether events have immutable IDs and timestamps, whether signatures cover the raw body, expected volume, ordering guarantees, and the business state machine. Assume at-least-once delivery, 10,000 events per second, and no global ordering.

The Naive Design and Why It Fails

Checking a process-local set is lost on restart and differs across replicas. Acknowledge-before-persisting loses events when the worker crashes. Applying every event in arrival order can regress state when an older event arrives late.

Better Design

Verify the signature and timestamp, insert provider + event ID into a dedupe table with a unique key, and process the event in the same transaction as the business transition. Acknowledge after the durable commit; return success for an already-seen event.

flowchart TD
    A[Webhook request] --> B[Verify signature and timestamp]
    B --> C[Unique event ID insert]
    C -->|duplicate| D[Return 2xx]
    C -->|new| E[Apply state transition]
    E --> F[Commit and enqueue outbox work]
    F --> G[Return 2xx]

Ordering and Failure Handling

Store event version or sequence and reject transitions that move backward. If an event arrives before its predecessor, persist it as pending and reconcile from the provider. A dead-letter queue holds invalid or repeatedly failing events with the raw payload, error, and attempt metadata; replay uses the same event ID.

Inbox Schema and State Transitions

The dedupe record should describe processing, not merely prove that a request was seen:

CREATE TABLE webhook_inbox (
    provider      text NOT NULL,
    event_id      text NOT NULL,
    resource_id   text NOT NULL,
    event_version bigint,
    status        text NOT NULL,
    payload_hash  text NOT NULL,
    received_at   timestamptz NOT NULL,
    processed_at  timestamptz,
    error_code    text,
    PRIMARY KEY (provider, event_id)
);

Insert the inbox row and update the business aggregate in one local transaction when both use the same database. If the event is already PROCESSED, acknowledge it without repeating the side effect. If it is RETRYABLE, a worker may claim it with a lease; if it is FAILED, keep it visible for an operator or bounded replay. Store a hash so a malicious or misconfigured sender cannot reuse an event ID with a different payload.

For a resource-level state machine, compare the event version before applying it. An event for PAID at version 8 must not move an order back to AUTHORIZED at version 7. When a provider has no sequence number, use its documented creation time only as a hint and reconcile from the provider API; local arrival time is not an ordering guarantee.

Authentication, Replay, and Recovery

Verify the signature over the exact raw body before parsing or queueing the event. Enforce a timestamp tolerance, support secret rotation through a key ID, and keep a bounded replay cache for signed delivery IDs. Signature validation proves origin, not freshness: a captured valid request can still be replayed unless the event ID and timestamp are checked.

At 10,000 events per second, do not perform slow provider calls inside the HTTP handler. Persist the inbox row, enqueue an internal command, and return 2xx after the durable commit. A worker can then retry transient failures with backoff, preserve the original event ID, and send poison payloads to a DLQ. Replays should be selected by provider, resource, error class, and time range, with a dry-run count before side effects resume.

Measure duplicate rate, signature failures, inbox insert conflicts, processing age, pending predecessor age, DLQ depth, and reconciliation mismatches. Alert on the oldest unprocessed event and on a jump in duplicate delivery; a flat event count can hide a growing delay.

Scaling and Trade-offs

Partition consumers by account or resource ID when per-entity ordering matters. At 10,000 events/second, index the dedupe table by event ID and archive records after the provider retry window, retaining an audit reference. The database transaction adds latency but gives a clear business invariant; an asynchronous handler improves ingress availability but delays state visibility.

Interview Follow-Ups

1. Why acknowledge after processing?

Acknowledging first tells the provider the event is safe even if the process crashes before persistence. A later retry is preferable to silent loss. If processing is slow, persist an inbox record quickly, acknowledge, and process from the durable queue.

2. How do you verify signatures?

Use the raw request body, provider-specific signing secret, a constant-time comparison, and a timestamp tolerance to prevent replay. Rotate secrets with a key ID. Never trust an event merely because it came from an allowlisted IP.

3. What if two replicas process the same event?

The unique constraint lets one insert win; the other treats the conflict as a duplicate and returns success after confirming the winner’s state. Keep the business update in the same transaction as the inbox insert. This makes replicas converge without a distributed mutex.

4. How do you handle out-of-order events?

Use a versioned state machine and compare event sequence or creation time. Buffer events whose prerequisite is missing and reconcile after a bounded delay. Do not use wall-clock time alone when the provider offers a stronger sequence.

5. How do you replay safely?

Fix the consumer, select a bounded DLQ range, and replay with original event IDs. The dedupe record must distinguish an already-successful event from one that failed before completion. Measure replay rate and downstream side effects during the exercise.

Review duplicate payments, Kafka offsets, and observability.

References

  1. Stripe: Webhook best practices
  2. AWS: Event-driven architecture

YouTube Videos

  1. “Webhook Idempotency and Reliability” - Hussein Nasser [https://www.youtube.com/watch?v=DOFflggE_0Q]

Share this post on:

Next in Series

Continue through the [object Object] with the next recommended article.

Related Posts

Keep Learning with New Posts

Subscribe through RSS and follow the project to get new series updates.

Was this guide helpful?

Share detailed feedback

Previous Post
How Would You Prevent Duplicate Payments?
Next Post
Kafka Consumer Lag Keeps Growing: What Do You Do?