
An order service commits an order, the inventory service reserves stock, and payment rejects the charge. A local database transaction can make each service atomic, but it cannot automatically roll back a commit in another database. Distributed transactions require an explicit coordination and recovery contract.
This guide explains the boundary of local ACID, the transactional outbox, two-phase commit, Sagas, compensation, and reconciliation. Continue from ACID properties, two-phase commit, and the Saga pattern.
Table of Contents
Open Table of Contents
Where Local ACID Stops
ACID protects operations within one database transaction. A service can atomically update its order row and outbox record, but a remote HTTP call is outside that transaction. Network timeouts create unknown outcomes: the remote operation may have committed even though the caller saw an error.
flowchart TD
A[Order request] --> B[Order DB transaction]
B --> C[Order committed]
B --> D[Outbox committed]
D --> E[Event publisher]
E --> F[Inventory local transaction]
F --> G[Payment local transaction]
G -->|Failure| H[Compensation or reconciliation]
Coordination Options
| Pattern | Atomicity | Availability | Best fit |
|---|---|---|---|
| Single database transaction | Strong within one store | High if store is healthy | Tightly related data |
| Transactional outbox | Atomic local state + event publication | High; async delivery | Event-driven workflows |
| Two-phase commit | Atomic commit across participants | Can block on coordinator/participant | Controlled homogeneous resources |
| Saga | Local commits + compensation | Good for long workflows | Microservices and business processes |
Do not choose 2PC or Saga because a diagram looks complete. Ask whether the business needs all-or-nothing storage atomicity or a recoverable business workflow. Compensation may not restore the exact previous world; it may issue a refund, release stock, or mark a manual-review state.
Transactional Outbox
The outbox writes business state and an event in one local transaction. A publisher later reads unpublished rows and sends them to a broker. If publishing succeeds but marking the row fails, the publisher retries; consumers therefore need idempotency.
BEGIN;
INSERT INTO orders (id, status) VALUES ('ord-42', 'PENDING');
INSERT INTO outbox_events (id, type, aggregate_id, payload)
VALUES ('evt-42', 'OrderCreated', 'ord-42', '{"orderId":"ord-42"}');
COMMIT;
The outbox solves the database-to-broker dual-write gap. It does not make inventory and payment atomic; those services still need a Saga, idempotency, and reconciliation.
Saga and Compensation
A Saga executes local transactions in sequence. An orchestrator sends commands and records state, or services react to each other’s events in choreography. If a later step fails, compensating actions reverse the business effect where possible.
For an order, compensation could cancel a reservation, issue a payment refund, and mark the order CANCELLED. Each step must be idempotent, time-bounded, observable, and safe when messages arrive twice or out of order. Store a Saga ID and step version with every local effect.
Recovery and Reconciliation
Unknown outcomes require reconciliation, not blind retry. Query the provider by idempotency key, compare local and remote state, and move the workflow to a durable REQUIRES_REVIEW state when automatic repair is unsafe. Metrics should show active Sagas, step age, compensation failures, and reconciliation backlog.
Interview Questions
1. Why does local ACID not span microservices?
Each service owns a separate transaction manager and database. A network call between them is not part of either local commit, and failures can occur after one side commits. Cross-service atomicity requires a protocol or a business workflow with recovery.
2. What does the transactional outbox solve?
It atomically records a business change and the event that describes it in one local transaction. A publisher later delivers the event with retries. It does not make downstream services commit atomically or eliminate duplicate delivery.
3. When is a Saga preferable to 2PC?
A Saga fits long-running, heterogeneous workflows where holding locks across services would hurt availability and latency. It accepts intermediate states and uses compensating actions. The business must tolerate eventual consistency and define what compensation means.
4. How do you handle a timeout after payment?
Treat the outcome as unknown, query the provider using the idempotency key, and reconcile the local order. Do not issue a second charge solely because the response timed out. Persist the workflow state and alert when reconciliation exceeds its SLO.
5. How do you make Saga steps safe to retry?
Use a stable Saga ID and step ID, conditional state transitions, unique constraints, and idempotent external requests. Record completion before acknowledging the command. Duplicate messages should return the existing result rather than perform a second side effect.
Conclusion
Distributed transactions are a correctness and recovery problem, not a magic annotation. Keep local state and events atomic with an outbox, use 2PC only where its blocking and resource assumptions fit, and use Sagas with explicit compensation for long workflows. Always design unknown outcomes, duplicate delivery, and reconciliation before the first failure.
References
- AWS: Saga orchestration pattern
- Microsoft: Saga design pattern
- microservices.io: Transactional outbox
YouTube Videos
- “Distributed Transactions Explained: 2 Phase Commit vs Saga Pattern” - Hello Interview [https://www.youtube.com/watch?v=DOFflggE_0Q]