Distributed Transactions Across Services: The Saga Pattern

1. Why ACID fails at service boundaries

A classic database transaction guarantees atomicity within a single database: either all changes get committed, or none do. As soon as a business operation, say an order, simultaneously reduces stock in the inventory system, authorizes a payment in the payment service, and creates a shipping request in the logistics service, that guarantee no longer applies, because each service owns its own database. Distributed transactions across services need to solve this problem without merging the individual services' databases.

The obvious but problematic approach is two phase commit across all involved databases. This mechanism holds locks on all involved resources for the entire duration of the operation, which in a distributed system with independently deployable services leads to massive availability problems: if a single service fails while locks are held, all other participating services block until the problem is resolved manually.

The saga pattern solves this dilemma differently: instead of a single atomic operation across all services, a saga consists of a sequence of independent, local transactions, each running within its own service with its own ACID guarantee. If a step fails, already completed steps get undone through explicit compensation actions, instead of relying on a global lock that simply does not exist in this model.

2. The saga pattern as an alternative to global locks

A saga models a business operation as an ordered sequence of local transactions T1, T2, T3 through Tn, each running in a different service. Each transaction Ti has a corresponding compensating transaction Ci that undoes the effect of Ti if a later step in the chain fails. For an order this could mean: T1 reserves stock, T2 authorizes payment, T3 creates a shipping order. If T3 fails, C2 runs (cancel payment authorization) followed by C1 (release stock reservation).

The central conceptual difference from classic transactions: during a saga, intermediate states are visible to other parts of the system. Between T1 and T2 there is a state where stock is reserved but payment is not yet authorized, a state that would never be visible externally in a classic transaction. This visibility of intermediate states is the price distributed transactions pay in the saga model, in exchange for avoiding global locks.

Sagas do not guarantee isolation in the classic ACID sense. Two concurrently running sagas can theoretically interfere with each other, for example if both want to reserve the same scarce stock. This lack of isolation must be compensated for through application specific measures, such as pessimistic reservations at the service level, or by explicitly accepting occasional conflicts that then get resolved through compensation.

3. Choreography vs. orchestration of sagas

Two fundamental implementation styles exist for sagas. With choreography, each service reacts to events triggered by other services and in turn triggers new events, without a central control instance. The order service triggers an "OrderCreated" event, the payment service reacts with a payment authorization and triggers "PaymentAuthorized", the logistics service reacts to that in turn. This decoupling is elegant for simple flows but becomes hard to follow for complex sagas with many conditional branches, because the overall flow is spread across many service implementations.

With orchestration, a central saga orchestrator takes explicit control: it calls each service in the correct order, waits for results, and decides on failure which compensation steps to trigger in which order. The flow is visible in a single place in the code, which significantly simplifies debugging and traceability, at the cost of an additional, central component that itself needs to run highly available.

For most production systems with more than three or four involved services, orchestration is recommended, because explicit flow control makes complexity more manageable once error handling and conditional logic come into play. Choreography suits simple, linear flows with few participants well, where decoupling brings more benefit than the missing central overview costs.

-- Example: tracking saga state in an orchestrator table
CREATE TABLE saga_instances (
    saga_id UUID PRIMARY KEY,
    saga_type VARCHAR(100) NOT NULL,
    current_step VARCHAR(100) NOT NULL,
    status VARCHAR(20) NOT NULL, -- RUNNING, COMPLETED, COMPENSATING, FAILED
    payload JSONB NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Update the progress of a saga
UPDATE saga_instances
SET current_step = 'payment_authorized',
    status = 'RUNNING',
    updated_at = now()
WHERE saga_id = '3f29a1c4-...';

-- Identify failed sagas for compensation
SELECT saga_id, current_step, payload
FROM saga_instances
WHERE status = 'FAILED'
ORDER BY updated_at ASC;

4. Designing compensation logic correctly

A compensating transaction is not a true rollback in the database sense, but a semantic reversal. A payment authorization is not deleted but offset by a reversal entry, because the original entry must be preserved for audit and compliance reasons. This semantic rather than technical reversal is a fundamental difference from rolling back a classic transaction and must be explicitly considered when designing every compensation step.

Not every action is compensable. An email notification that has already been sent to a customer cannot be pulled back. For such non compensable steps, the rule is to place them as late as possible in the saga, ideally only after all other, compensable steps have completed successfully. A confirmation email should therefore be the last step of an order saga, not one of the first.

Compensation steps must themselves be robust against failures, because a failing compensation step is an even harder problem to fix than the original error. In practice this means: equip compensation logic with its own retry mechanisms and route cases where compensation itself repeatedly fails into a manual handling queue, instead of silently discarding them.

5. The outbox pattern for reliable events

A subtle but common problem with choreographed sagas is the consistency between a local database change and sending the corresponding event. If a service first writes to its database and then sends an event to a message broker, a failure can occur between these two steps: the database change is committed, but the event was never sent because the process crashed beforehand. Other services never learn about the change, and the saga gets stuck in an inconsistent intermediate state.

The outbox pattern solves this problem by making event creation part of the same local database transaction as the actual business data change. Instead of sending the event directly to the broker, an entry gets written to an outbox table within the same transaction. A separate process continuously reads this outbox table and reliably publishes the events to the message broker, with retry logic for transmission failures.

This reduces the atomicity between database change and event dispatch to the local, already existing ACID guarantee of the individual service database, without needing distributed transactions across database and message broker. The outbox pattern is thus a central building block for reliable distributed transactions in both choreographed and orchestrated sagas alike.

-- Outbox table: event creation as part of the same local transaction
CREATE TABLE outbox_events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_type VARCHAR(100) NOT NULL,
    aggregate_id VARCHAR(100) NOT NULL,
    event_type VARCHAR(100) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at TIMESTAMPTZ
);

-- Both writes inside the same ACID transaction
BEGIN;
UPDATE orders SET status = 'confirmed' WHERE id = 4821;
INSERT INTO outbox_events (aggregate_type, aggregate_id, event_type, payload)
VALUES ('order', '4821', 'OrderConfirmed', '{"order_id": 4821}'::jsonb);
COMMIT;

-- Separate publisher process reads unpublished events
SELECT id, event_type, payload
FROM outbox_events
WHERE published_at IS NULL
ORDER BY created_at ASC
LIMIT 100;

6. Idempotency as a prerequisite for sagas

Because networks are unreliable and message brokers deliver messages at least once, occasionally even multiple times, every step of a saga must be idempotent: executing the same action multiple times must not produce a different result than executing it once. A payment authorization that charges the amount twice on duplicate delivery is a classic example of missing idempotency with direct financial consequences.

The usual mechanism for idempotency is a unique idempotency key sent along with every request. The receiving service checks before execution whether this key has already been processed and, on a repeat, returns the previously computed result instead of executing the action again. This key should be stored in a dedicated table with a unique constraint, so that even simultaneous, duplicate requests do not create a race condition.

Idempotency must also apply to compensation steps, not just the original action. A compensation step that mistakenly cancels twice on retry can lead to incorrect states just as easily as a non idempotent original action. This symmetry is often overlooked in practice, because developers test the compensation path less frequently than the success path of a saga.

7. Making eventual consistency visible in the application

Sagas by definition only deliver eventual consistency: once all steps complete, the system is consistent, but during execution, intermediate states exist that may be visible to other parts of the system. An application that hides this difference from users produces confusing experiences: an order shown in the UI as "confirmed" even though payment authorization has not yet completed may later need to be canceled through compensation, which feels unexpected and hard to follow for the user.

A proven practice is to explicitly represent the saga status in the data model and the user interface, for example as "processing", instead of pretending the operation has already fully completed. This makes eventual consistency understandable for users, instead of hiding it and later revealing it through surprising corrections.

For internal systems that rely on consistent data, such as reporting or analytics, it should be clearly defined how long after a saga completes consistency can be expected. This time span depends on the number of steps, the reliability of the message broker, and the retry configuration, and should be measured, not assumed.

8. Monitoring and debugging distributed sagas

Distributed transactions across multiple services are significantly harder to debug than a single database transaction, because the relevant state is spread across multiple systems. Distributed tracing with a consistent correlation ID that propagates through every step of a saga is not an optional addition but a basic prerequisite for reconstructing the complete flow across service boundaries when something fails.

A dedicated saga status dashboard showing running, completed, and failed sagas is practically indispensable for operating production systems built on the saga pattern. Without this visibility, stuck sagas that were neither successfully completed nor fully compensated often go unnoticed for days, until a user complains about an inconsistent state.

Alerts should react to sagas lingering longer than expected in a "running" or "compensating" state. Such lingering usually points to a failed downstream service or faulty compensation logic that requires manual intervention before the affected record remains permanently in an inconsistent state.

9. The saga pattern compared to other approaches

The table below compares the saga pattern with alternative approaches for distributed transactions across services and shows when each approach makes sense.

In practice, the key insight is that distributed transactions across services only make sense when splitting into separate services is actually necessary for business or organizational reasons. Where a monolithic system with a single database suffices, the added complexity of sagas, compensation, and the outbox pattern is simply not justified.

10. Summary

Distributed transactions across services require a fundamentally different approach than classic ACID transactions. The saga pattern replaces global locks with a chain of local transactions and explicit compensation logic for the failure case. The choice between choreography and orchestration depends on the complexity of the flow, while the outbox pattern ties reliable event delivery to each service's local ACID guarantee.

Idempotency is not a minor detail but a basic prerequisite for every saga, both for the original action and for compensation steps. Anyone who combines these building blocks cleanly and makes eventual consistency visible for users and operations, instead of hiding it, achieves consistent business workflows across service boundaries without the availability problems of classic distributed transaction mechanisms such as two phase commit.

11. FAQ: Distributed Transactions Across Services