
The order row is committed, inventory is held, and the payment provider returns an error. The answer is not a cross-service database transaction by default; it is an explicit workflow with compensation and a customer-visible state. See Saga pattern, two-phase commit, and duplicate payments.
The Interview Question
“Order creation succeeds, payment fails or times out. How do you recover without losing the order, leaking inventory, or charging twice?”
Clarifying Questions
Ask whether payment authorization can be queried, whether inventory holds expire, whether the customer may retry, and which actions are reversible. Assume local ACID per service, at-least-once events, a ten-minute inventory hold, and an uncertain provider timeout.
The Naive Design and Why It Fails
A synchronous chain that rolls back an order after a timeout cannot know whether payment committed. A distributed lock does not make two databases atomic. Deleting the order immediately can lose the audit trail and confuse the customer.
Better Design
Persist an order state machine: PENDING_PAYMENT, PAID, PAYMENT_UNKNOWN, CANCELED, and REFUND_PENDING. A Saga orchestrator sends idempotent commands, while an outbox publishes transitions after each local commit. Query uncertain payments before retrying.
flowchart TD
A[Create order and hold stock] --> B[Authorize payment]
B -->|success| C[Confirm order]
B -->|decline| D[Cancel and release stock]
B -->|timeout| E[Payment unknown]
E --> F[Query provider]
F -->|captured| G[Confirm or refund]
F -->|not found| H[Safe retry with same key]
Make the State Machine Explicit
Do not represent every ambiguous outcome as FAILED. A useful transition table separates what the system knows from what it is still investigating:
| Current state | Event | Next state | Action |
|---|---|---|---|
PENDING_PAYMENT | authorization succeeds | PAID | Confirm the order and release the payment workflow. |
PENDING_PAYMENT | definitive decline | CANCELED | Release the inventory hold and show a retry option. |
PENDING_PAYMENT | timeout | PAYMENT_UNKNOWN | Stop new charges and query the provider by the same key. |
PAYMENT_UNKNOWN | provider says captured | PAID or REFUND_PENDING | Confirm if stock is still valid; otherwise compensate. |
PAYMENT_UNKNOWN | provider says absent | PENDING_PAYMENT | Retry once with the original idempotency key. |
| Any active state | hold expires | CANCELED or MANUAL_REVIEW | Never silently discard a paid order. |
Each transition should be conditional on the current version. A late decline must not overwrite a confirmed payment, and a retrying worker must not execute a compensation that another worker already completed.
Orchestration and Durable Commands
The orchestrator should not hold a database transaction while calling the payment provider. It commits a local command, publishes an outbox event, and lets a worker claim the command with a lease. A simplified loop looks like this:
claim order workflow where status is actionable and lease is expired
if state is PENDING_PAYMENT:
call provider with order_id as the idempotency key
apply one compare-and-set transition
publish the resulting event through the outbox
if state is PAYMENT_UNKNOWN:
query provider before creating any new intent
renew lease only while the worker is making bounded progress
The lease prevents a crashed worker from holding the workflow forever, while the state version prevents an old worker from applying a late result. Commands and events need stable IDs so retries are observable and idempotent.
Reconciliation Is Part of the Design
At 2,000 orders per second, even a one-in-ten-thousand ambiguous payment produces roughly 12 unresolved intents per minute. A reconciliation job should scan by age, query the provider or settlement file, compare local order, payment, and inventory states, and emit a repair command. Keep the original provider reference, request hash, response code, and timestamps so an operator can explain the decision.
Compensation is not always immediate. If stock has already shipped, refunding the payment may be safer than trying to reverse fulfillment. If a payment is captured but the customer has no order, create a support-visible exception rather than automatically issuing multiple refunds. The business policy determines the compensating action; the technical design makes it durable and repeatable.
Failure Handling and Reconciliation
Every command carries an order and idempotency key. A worker resumes stale states, and a reconciliation job compares provider settlement reports, orders, and inventory holds. Compensation is a business action: release stock, issue a refund, or route to manual review—not a pretend rollback.
Scaling and Trade-offs
Partition workflow work by order ID, use a durable queue, and cap provider retries. At 2,000 orders/second, retain state transitions and provider references for audit and replay. The Saga improves availability and service autonomy but allows temporary inconsistency; strict atomicity may require a different boundary.
Interview Follow-Ups
1. Why not use two-phase commit?
2PC can coordinate databases but holds prepared resources, blocks during coordinator failure, and often does not include an external payment provider. A Saga lets each service commit locally and defines compensation. Choose 2PC only when participants support it and atomicity outweighs availability.
2. What does the customer see during an unknown payment?
Show a pending payment status and prevent a second intent for the same order. Notify the customer when reconciliation resolves it. Never label an unknown result as failed merely because the HTTP request timed out.
3. How do you prevent duplicate compensation?
Use a unique command key and durable state transitions. A retry of “release inventory” should return the original result. Monitor compensation age and send exhausted work to an operator queue.
4. What if inventory expires before payment resolves?
The state machine must define whether the order can be canceled, re-reserved, or manually fulfilled. Do not silently charge without stock. A successful payment after expiry may require an immediate refund or alternate fulfillment.
5. What metrics indicate the workflow is unhealthy?
Track duration by state, unknown payments, compensation failures, stuck holds, provider timeout rate, replay count, and reconciliation differences. Alert on age of the oldest pending workflow, not only error percentage.
Related Practice
Continue with payment system design, webhook deduplication, and distributed transactions.
References
YouTube Videos
- “Saga Pattern and Distributed Transactions” - Gaurav Sen [https://www.youtube.com/watch?v=DOFflggE_0Q]