ADevGuide 
Polling a payment provider every few seconds is slow, expensive, and still leaves a delay between a charge completing and your application learning about it. A webhook reverses that flow: the provider pushes an HTTP event to your endpoint. The reliability challenge is that the receiver may be slow, unavailable, or process the same event more than once.
This guide explains how to design secure, retry-safe webhook delivery for backend interviews. It builds on idempotency in REST APIs and API rate limiting in the Backend Developer Interview Guide.
Table of Contents
Open Table of Contents
What a Webhook Is
A webhook is an HTTP callback sent when an event occurs. A payment provider can send payment.succeeded; a source-control platform can send pull_request.opened; an email provider can send message.bounced. The producer owns delivery, while the consumer owns a public endpoint, signature verification, and event processing.
Treat delivery as at-least-once, not exactly once. The sender cannot know whether a timeout happened before or after your business logic ran, so retrying is safer than silently losing the event. That means every receiver must tolerate duplicates, delayed events, and often out-of-order events.
Delivery Architecture
The producer should persist an event before attempting external delivery. A durable outbox or event table lets a worker retry independently of the user-facing request that caused the event.
flowchart TD
A[Business event commits] --> B[(Outbox or event store)]
B --> C[Webhook dispatcher]
C --> D[Sign payload with timestamp and event ID]
D --> E[Subscriber endpoint]
E --> F{2xx before timeout?}
F -->|Yes| G[Record delivery success]
F -->|No| H[Schedule backoff retry]
H --> I{Retry budget exhausted?}
I -->|No| C
I -->|Yes| J[Dead-letter queue and alert]
E --> K[Verify signature and timestamp]
K --> L[Atomically record event ID]
L --> M[Atomically persist event record and outbox job]
M --> N[Return 2xx quickly]
M --> O[Worker processes durable job]
classDef flow fill:#e8f0fe,stroke:#1a73e8,stroke-width:2px,color:#000000;
class A,B,C,D,E,F,G,H,I,J,K,L,M,N,O flow;The receiver should do minimal synchronous work: read the raw request body, verify the signature, then atomically persist both the deduplicated event record and an internal outbox job in one database transaction before returning a 2xx. If those are separate operations, a crash after recording the event but before queue persistence makes a retry look like a duplicate and loses the business work. A worker recovers accepted or stuck processing records from the durable job table. Long processing in the HTTP handler increases timeouts and causes needless redelivery. GitHub, for example, recommends acknowledging deliveries within ten seconds.
Retries and Delivery Guarantees
Retry transient failures such as connection errors, timeouts, 408, 429, and most 5xx responses with exponential backoff and jitter. Do not retry most 4xx responses: a malformed payload or an unauthorized endpoint usually needs a configuration fix, not another request. Apply a finite attempt and time budget, record every attempt, and send exhausted deliveries to a dead-letter queue or dashboard for replay.
Per-subscriber queues and concurrency limits prevent one unhealthy customer endpoint from consuming every worker. Respect Retry-After when a receiver explicitly throttles you. Delivery success means only that the receiver acknowledged the HTTP request; it does not prove its downstream database or business process eventually succeeded. That distinction is central in an interview answer.
Verify Signatures Before Processing
Use HTTPS, but do not mistake TLS for sender authentication. Sign the exact raw body plus timestamp and event ID with an HMAC secret or an asymmetric private key. The consumer reconstructs the signed message, calculates the expected signature, and uses a constant-time comparison. Parse JSON only after verification if the provider’s signature scheme covers raw bytes.
Reject timestamps outside a small allowed window to reduce replay risk, rotate secrets with an overlap period, and never log signing secrets or full sensitive payloads. IP allowlists can add defense in depth but are brittle and should not replace cryptographic verification. An API gateway can route and protect the public endpoint, but durable delivery and business deduplication still belong in the webhook system.
Consumer Idempotency and Ordering
Use the producer’s stable event ID as the deduplication key. Insert it into a table with a unique constraint before performing side effects; a concurrent duplicate then becomes a safe no-op. Store enough metadata to audit the event, including event type, source, received time, and processing state. This is the same atomic claim pattern used for a payment idempotency key.
Do not assume delivery order. A retry of an older event can arrive after a newer one, and separate workers can complete out of order. Make handlers idempotent and version-aware: for a subscription.updated event, compare the event version or fetch current state from the producer before overwriting local data. When ordering is truly required, partition by entity and serialize work for that entity, but keep the version or sequence guard: serialization preserves enqueue order, not the producer’s original event order. This trades parallelism and buffering complexity for stronger correctness.
For implementation context, see background jobs and the broader Backend category. Incoming webhook endpoints also need bounded request sizes and appropriate rate limits so a delivery spike cannot exhaust application capacity.
Observability and Operations
Track delivery attempts, status codes, latency, queue depth, retry age, success rate, and dead-letter count per subscriber. Give producers a delivery log with redacted request metadata and a safe replay control. Give consumers enough event IDs and correlation IDs to connect a business state change to the callback that triggered it.
Test the unhappy path deliberately: timeout after accepting the event, duplicate delivery, a forged signature, stale timestamp, out-of-order updates, and a receiver that returns 429. A design that only works against a fast localhost callback is not ready for third-party integrations.
Interview Questions
1. Why should a webhook consumer return 2xx before doing all business processing?
The sender’s timeout is usually much shorter than a real business workflow. If the handler waits for an email, database migration, or downstream API call, the sender can time out and deliver the same event again even though the work later completes. I would verify the signature, persist and deduplicate the event, enqueue the expensive work, then acknowledge. This trades a queue and worker for predictable delivery behavior and better failure recovery.
2. How do you prevent duplicate webhook side effects?
I would use the provider’s stable event ID with a unique database constraint and atomically insert it before any side effect. A duplicate event then finds an existing record and exits without sending another email or changing the order twice. The record needs a processing state so a crash between acceptance and completion can be retried safely. This is at-least-once delivery with idempotent consumption, not an unsupported exactly-once promise.
3. How do you verify a webhook is authentic?
I would verify an HMAC or asymmetric signature over the exact raw body and signed metadata using the provider’s documented scheme. A timestamp freshness check limits replay attacks, and constant-time comparison avoids leaking signature information. HTTPS protects transport but does not by itself prove that the request came from the expected provider. Secret rotation, payload-size limits, and redacted logs round out the operational controls.
4. Which failures should a webhook producer retry?
Network failures, timeouts, 429, and most 5xx responses are reasonable retry candidates because they often indicate a temporary receiver or network problem. I would use exponential backoff with jitter, a finite budget, and per-subscriber concurrency limits. Most 4xx responses should stop and surface a configuration problem rather than create retry noise. The policy must be documented because receivers build their idempotency and alerting around it.
5. How do you handle events that arrive out of order?
First, I would make each handler idempotent because retries alone already break ordering assumptions. Then I would use an event version, sequence number, or source-of-truth fetch to avoid applying an older state over a newer one. If strict per-entity ordering is required, partition the queue by entity ID and process one partition serially. That improves correctness but reduces parallelism, so I would reserve it for workflows that truly need it.
Conclusion
- Webhooks are push callbacks that need an explicit at-least-once delivery contract.
- Durable outbox records, bounded retries, and dead-letter handling make producer delivery recoverable.
- Consumers should verify raw-body signatures, reject stale replays, deduplicate event IDs, and acknowledge quickly.
- Ordering is not guaranteed; use versions or entity serialization only when the product requires it.
- Delivery logs, replay controls, and failure-path tests are production requirements, not extras.
The next topic in this series refreshes CORS for backend developers - the browser policy that controls which web applications may call an API. Revisit idempotency for the shared pattern that makes duplicate delivery safe.
References
- Best Practices for Using Webhooks - GitHub Docs https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks
- Standard Webhooks Specification - Standard Webhooks https://github.com/standard-webhooks/standard-webhooks/blob/main/spec/standard-webhooks.md
YouTube Videos
- “Webhook VS API | What is Webhooks | Explained in detail with example” https://www.youtube.com/watch?v=GND6y6lolZw