Idempotency in REST APIs: How to Prevent Duplicate Payments

A customer taps “Pay” once, the server creates the charge, and the mobile connection drops before the response arrives. The customer taps again. Without an idempotency contract, that ordinary timeout can become two charges, two orders, and a costly reconciliation problem.

This guide explains how to make retry-prone writes safe with an idempotency key, a durable record, and an atomic decision. It builds on REST API design and API rate limiting in the Backend Developer Interview Guide.

Table of Contents

What Idempotency Means

An operation is idempotent when repeating the same intended operation does not create additional intended side effects. HTTP GET, PUT, and DELETE are defined around idempotent server-side effects, although their responses may differ. A POST /payments is not inherently idempotent: the server normally treats each request as a new instruction.

For a payment, order, refund, or provisioning endpoint, clients need permission to retry after an uncertain failure. An Idempotency-Key turns one logical user action into a stable identity that the server can recognize. It does not make every POST safe automatically; the server must persist and enforce the contract.

The Lost-Response Failure

The risky case is not a request that definitely failed. It is an outcome the client cannot observe:

1. The client sends POST /payments with key pay-8c1....
2. The API successfully authorizes the card and creates payment p_123.
3. The response is lost because of a timeout, app restart, or network change.
4. The client retries because it cannot distinguish “server failed” from “response was lost.”

If the retry creates p_124, the system has charged twice. Retrying with the same idempotency key instead lets the API return the already-established result for p_123. This is why a client must create one high-entropy key per logical action, then reuse it only for retries of that action.

The Idempotency-Key Contract

Make the rule explicit in API documentation and SDKs. A typical request looks like this:

POST /v1/payments HTTP/1.1
Authorization: Bearer <token>
Idempotency-Key: 8c1d4432-8c93-4e3e-8f22-a4117d8f1b0d
Content-Type: application/json

{
  "orderId": "order_742",
  "amount": 4999,
  "currency": "USD"
}

Scope the key to the authenticated tenant and operation, for example (account_id, POST:/v1/payments, key). Bind it to a request fingerprint made from canonical request fields. A subsequent request with the same scope, key, and fingerprint replays the original status and body. The same key with a different amount or order must fail clearly, commonly with 409 Conflict, rather than silently return an unrelated payment.

Request Flow

flowchart TD
    A[Client sends payment with Idempotency-Key] --> B[Authenticate and validate request]
    B --> C[Atomic insert or compare-and-set for key and fingerprint]
    C --> D{Claim outcome?}
    D -->|New key| E[Create pending record]
    E --> F[Create payment]
    F --> G[Store completed response]
    G --> H[201 Created]
    D -->|Completed and fingerprint matches| I[Replay stored response]
    I --> H
    D -->|Fingerprint differs| J[409 Conflict]
    D -->|In progress| K[Return in-progress response or wait briefly]
    H --> L{Client received response?}
    L -->|No, retry same key| A
    L -->|Yes| M[Finish]
    J --> M
    K --> M

    classDef flow fill:#e8f0fe,stroke:#1a73e8,stroke-width:2px,color:#000000;
    class A,B,C,D,E,F,G,H,I,J,K,L,M flow;

The key must be claimed before the irreversible side effect. Otherwise, two concurrent retries can both see no record and both create a charge. The exact response for an in-progress duplicate is a product decision: return 409 or 202, or wait for the original request for a bounded period. The important part is that it does not perform the business action twice.

Store the Result and Make the Claim Atomic

Use a durable table or key-value record whose uniqueness is enforced by the same system that protects the business write. A relational example is:

CREATE TABLE idempotency_records (
  account_id UUID NOT NULL,
  operation TEXT NOT NULL,
  idempotency_key TEXT NOT NULL,
  request_fingerprint TEXT NOT NULL,
  status TEXT NOT NULL, -- pending, completed, or failed
  response_status INTEGER,
  response_body JSONB,
  expires_at TIMESTAMPTZ NOT NULL,
  PRIMARY KEY (account_id, operation, idempotency_key)
);

Attempt an INSERT first; let the unique constraint decide which concurrent request owns the work. The winner creates the payment and records the result in the same transactional boundary where possible. A duplicate reads the record, compares its fingerprint, and either returns the stored response or reports a conflict.

Storing the full response makes replay exact and simple, but it consumes more storage and can preserve stale representations. Storing only payment_id is cheaper, but rebuilding the response later must not expose changed state as if it were the original result. Choose based on the endpoint’s response stability, audit needs, and retention window. The same atomicity principle appears in database transactions and message consumers that handle at-least-once delivery.

Key Reuse, Expiry, and Failure Policy

Reject a key reused with a changed request. Without fingerprint validation, a buggy client could accidentally attach an old payment key to a new order and receive a misleading success response. Also validate key size and characters; never concatenate untrusted keys into database queries or cache commands.

Keep records long enough to cover the client’s legitimate retry horizon and payment-provider behavior, then expire them deliberately. A short TTL reduces storage but risks treating a late retry as a new payment. A long TTL gives stronger duplicate protection but needs retention and privacy planning. For failed requests, document whether a retry with the same key replays the failure or may try again; transient failures usually need a state model that does not permanently poison a key before the outcome is known.

A pending record also needs a recovery path. A worker can successfully call a payment provider and crash before it writes completed locally. After a bounded lease expires, a recovery worker or retry should query the provider using the same provider-side idempotency key, then finalize the local record from that result. Do not simply unlock and run the charge again: that converts an operational crash into a duplicate payment.

Idempotency and rate limiting solve different failures. Rate limiting controls how much work a caller can request; idempotency prevents repeated delivery of one logical write from producing repeated side effects. For the broader API path, browse the Backend category.

Interview Questions

1. Why is an idempotency key needed if the database has transactions?

A database transaction protects a local set of writes from partial commit, but it cannot tell a retried HTTP request that its first attempt already committed. The request may have completed before the client timed out, or it may have triggered an external payment provider outside the database. The idempotency record gives the business action a durable identity across requests. I would combine it with transactions where possible, then use provider-side idempotency and reconciliation for external effects.

2. How do you prevent two identical requests from winning concurrently?

I would put a unique constraint on the tenant, operation, and idempotency key, then use an atomic insert or compare-and-set to claim the key. A read-then-write check is unsafe because two workers can both read “missing” before either inserts. The winner owns the business operation; the loser reads the pending or completed record. This trades a little coordination and storage for preventing a far more expensive duplicate charge.

3. What happens if a client reuses a key with a different payload?

The server should compare a canonical request fingerprint and reject the mismatch rather than replaying a previous response. Returning a payment for the wrong amount is dangerous, while executing the changed request defeats deduplication. I would return a documented conflict error that tells the client to create a new key for a new action. The fingerprint should include only the fields that define the operation and should be scoped to the authenticated account.

4. Should completed idempotency records live forever?

Usually no. The retention period should cover realistic retries, asynchronous completion, and any provider-specific duplicate window. Retaining records forever increases storage cost and may keep sensitive response data longer than necessary; deleting them too early reopens the duplicate-payment risk. I would document the TTL, measure late retries, and store minimal response data when possible. Financial systems may also retain a separate immutable business audit record for much longer.

5. Does an idempotency key provide exactly-once processing?

No. It provides an at-most-once effect for a defined operation scope and retention window when the implementation is correct. External providers, asynchronous workers, and regional failures can still create uncertain states that require provider idempotency, outbox records, reconciliation, and compensating actions. In an interview, I would avoid promising exactly once unless the whole end-to-end boundary can enforce it. The useful guarantee is safe retry behavior for the caller.

Conclusion

1. Idempotency makes uncertain network retries safe for one logical write.
2. A client reuses one high-entropy key only for retries of the same action.
3. Atomic key claiming and request fingerprints prevent concurrent and mismatched duplicates.
4. Persisted responses make duplicate retries predictable, while TTLs control storage and risk.
5. Idempotency complements transactions, provider controls, and reconciliation; it does not create universal exactly-once delivery.

The next topic in this series covers offset, cursor, and keyset pagination - how to read changing datasets predictably. Revisit API rate limiting for the related retry and backoff decisions at the API boundary.

References

1. Idempotency-Key Header - MDN Web Docs https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Idempotency-Key
2. Idempotency - Adyen Docs https://docs.adyen.com/development-resources/api-idempotency/

YouTube Videos

1. “Idempotency Keys: Preventing Double Charges in Payment APIs” https://www.youtube.com/watch?v=EpnxEsH5v5w