05 - Saga Pattern: Distributed Transactions with Choreography and Orchestration

One purchase, three services, no transaction covering them

In a monolith the purchase is a transaction: you create the order, decrement the stock, record the payment, and if anything goes wrong the database undoes everything. With three services, each with its own database, a BEGIN ... COMMIT that crosses them does not exist. The classic protocol for doing it is two-phase commit: a coordinator asks everyone to prepare, then orders the commit. It works, but it keeps the participants' resources locked until the coordinator decides, and it wants them all reachable at the same moment: two conditions that in independent services, with third-party APIs in the middle, you cannot guarantee.

The saga gives up global atomicity and replaces it with a sequence of local transactions, each committed immediately, plus for each one a compensating transaction that undoes its business effect if a later step fails. The concept comes from Hector Garcia-Molina and Kenneth Salem, who proposed it in 1987 for long-lived transactions in databases.

The price is that for a while the system is visibly half-done: the stock is decremented and the payment is not there yet. This article builds the saga of a purchase in two ways, with an orchestrator and with a choreography of events, runs them on the four cases that matter (success, refusal, temporary error, a compensation that fails) and says where each model hurts. It presupposes the vocabulary of events and commands.

The three services of the experiment and the idempotency key

The three classes below are the participants: Inventory, Payments and Orders, in memory. Every operation that modifies something receives an idempotency key (the saga id plus the name of the step) and remembers the ones already executed: receiving the same one twice does not repeat the effect. This is why, later on, a compensation delivered twice does not release the stock twice.

The two knobs transient and declines distinguish a temporary error (the gateway times out, worth retrying) from a definitive refusal (card declined, retrying is useless): the distinction decides whether you retry or compensate. unreachable makes the stock release fail, to prove what happens when it is the compensation that does not succeed.

export class TransientError extends Error {}

// Every operation carries an idempotency key: repeating it does not repeat the effect.
export class Inventory {
  stock = new Map<string, number>([['pen', 5]]);
  private done = new Set<string>();
  unreachable = 0; // number of calls to release() that will fail before one works

  reserve(key: string, product: string, q: number): void {
    if (this.done.has(key)) return;
    const available = this.stock.get(product) ?? 0;
    if (available < q) throw new Error('InsufficientStock');
    this.stock.set(product, available - q);
    this.done.add(key);
  }
  release(key: string, product: string, q: number): void {
    if (this.unreachable > 0) { this.unreachable--; throw new TransientError('Inventory unreachable'); }
    if (this.done.has(key)) return;
    this.stock.set(product, (this.stock.get(product) ?? 0) + q);
    this.done.add(key);
  }
}

export class Payments {
  transient = 0;      // temporary failures before success
  declines = false;   // definitive refusal (card declined)
  receipts = 0;
  private done = new Set<string>();
  pay(key: string): void {
    if (this.done.has(key)) return;
    if (this.transient > 0) { this.transient--; throw new TransientError('Payment gateway timeout'); }
    if (this.declines) throw new Error('PaymentDeclined');
    this.receipts++;
    this.done.add(key);
  }
}

export class Orders {
  statuses = new Map<string, string>();
  setStatus(id: string, status: string): void { this.statuses.set(id, status); }
}

This module has no output of its own: it is exercised by the two programs that follow. A note of realism: the memory of the keys already seen is a Set in RAM, so it is lost on restart. In production the key is saved in the same database and in the same transaction as the effect, otherwise idempotency protects only as long as the process does not die.

Orchestration: one place knows where the saga stands

In orchestration a central component knows the sequence of steps, invokes them one by one and, if one fails, runs the compensations of the steps already completed in reverse order. The reverse order is not a convention: the compensation of a step may depend on the later ones having already been undone.

In the program, runSaga keeps the list of completed steps and, at the first error, walks it backwards skipping the steps without a compensation (here pay, which if it fails has left nothing to undo, and confirm order, which is the last). withRetry retries only the TransientErrors and doubles the wait at each attempt; the wait is written in the log but not executed, so as not to slow down the experiment. The other errors go straight to compensation.

import { Inventory, Payments, Orders, TransientError } from './services.ts';

type Step = { name: string; run: () => void; compensate?: () => void; attempts?: number };
type Outcome = 'COMPLETED' | 'COMPENSATED' | 'BLOCKED';

// Repeats only the transient errors, with a doubled wait (here written to the log, not waited).
function withRetry(name: string, f: () => void, max: number, trace: string[]): void {
  let wait = 2;
  for (let t = 1; ; t++) {
    try { f(); return; } catch (e) {
      if (!(e instanceof TransientError) || t >= max) throw e;
      trace.push(`  ${name}: attempt ${t} failed, retrying in ${wait} s`);
      wait *= 2;
    }
  }
}

function runSaga(id: string, steps: Step[], trace: string[]): Outcome {
  const completed: Step[] = [];
  for (const s of steps) {
    try {
      withRetry(s.name, s.run, s.attempts ?? 1, trace);
      trace.push(`ok       ${s.name}`);
      completed.push(s);
    } catch (e) {
      trace.push(`FAILED   ${s.name}: ${(e as Error).message}`);
      // Compensation: only the completed steps, in reverse order.
      for (const c of [...completed].reverse()) {
        if (!c.compensate) continue;
        try { withRetry(`compensate ${c.name}`, c.compensate, 3, trace); trace.push(`compensated ${c.name}`); }
        catch (e2) { trace.push(`BLOCKED on compensate ${c.name}: ${(e2 as Error).message}`); return 'BLOCKED'; }
      }
      return 'COMPENSATED';
    }
  }
  return 'COMPLETED';
}

function scenario(title: string, setup: (inv: Inventory, pay: Payments) => void) {
  const inv = new Inventory(), pay = new Payments(), ord = new Orders();
  setup(inv, pay);
  const id = 'saga-1';
  const steps: Step[] = [
    { name: 'create order', run: () => ord.setStatus(id, 'PENDING'), compensate: () => ord.setStatus(id, 'CANCELLED') },
    { name: 'reserve stock', run: () => inv.reserve(`${id}:reserve`, 'pen', 2), compensate: () => inv.release(`${id}:release`, 'pen', 2) },
    { name: 'pay', run: () => pay.pay(`${id}:pay`), attempts: 3 },
    { name: 'confirm order', run: () => ord.setStatus(id, 'CONFIRMED') },
  ];
  const trace: string[] = [];
  const outcome = runSaga(id, steps, trace);
  console.log(`== ${title}\n${trace.join('\n')}\noutcome ${outcome} | order ${ord.statuses.get(id)} | pen stock ${inv.stock.get('pen')} | receipts ${pay.receipts}`);
}

scenario('all good', () => {});
scenario('payment declined', (_i, p) => { p.declines = true; });
scenario('payment: two timeouts, then ok', (_i, p) => { p.transient = 2; });
scenario('declined + inventory unreachable once', (i, p) => { p.declines = true; i.unreachable = 1; });
scenario('declined + inventory always unreachable', (i, p) => { p.declines = true; i.unreachable = 99; });

// The same compensation delivered twice does not release twice.
const inv = new Inventory();
inv.reserve('x:reserve', 'pen', 2);
inv.release('x:release', 'pen', 2);
inv.release('x:release', 'pen', 2);
console.log('== repeated compensation | pen stock', inv.stock.get('pen'), '(5 at the start)');

// Output:
// == all good
// ok       create order
// ok       reserve stock
// ok       pay
// ok       confirm order
// outcome COMPLETED | order CONFIRMED | pen stock 3 | receipts 1
// == payment declined
// ok       create order
// ok       reserve stock
// FAILED   pay: PaymentDeclined
// compensated reserve stock
// compensated create order
// outcome COMPENSATED | order CANCELLED | pen stock 5 | receipts 0
// == payment: two timeouts, then ok
// ok       create order
// ok       reserve stock
//   pay: attempt 1 failed, retrying in 2 s
//   pay: attempt 2 failed, retrying in 4 s
// ok       pay
// ok       confirm order
// outcome COMPLETED | order CONFIRMED | pen stock 3 | receipts 1
// == declined + inventory unreachable once
// ok       create order
// ok       reserve stock
// FAILED   pay: PaymentDeclined
//   compensate reserve stock: attempt 1 failed, retrying in 2 s
// compensated reserve stock
// compensated create order
// outcome COMPENSATED | order CANCELLED | pen stock 5 | receipts 0
// == declined + inventory always unreachable
// ok       create order
// ok       reserve stock
// FAILED   pay: PaymentDeclined
//   compensate reserve stock: attempt 1 failed, retrying in 2 s
//   compensate reserve stock: attempt 2 failed, retrying in 4 s
// BLOCKED on compensate reserve stock: Inventory unreachable
// outcome BLOCKED | order PENDING | pen stock 3 | receipts 0
// == repeated compensation | pen stock 5 (5 at the start)

In the case of the refusal the output shows the correct order: first the stock is released, then the order is cancelled, and in the end the stock is back to 5, the order is CANCELLED and the receipts are zero. In the case of the two timeouts the payment succeeds on the third attempt: had I compensated at the first error you would have cancelled a purchase that would have gone through.

The last case is the one to remember: when the stock release also fails three times, the saga ends in BLOCKED, with the order still PENDING and two pens taken out of the warehouse for a purchase that will not exist. No automatic mechanism resolves this situation: you need an explicit saga state that an alarm can read, and a human intervention or a queue of long retries. The last line shows idempotency: calling the same compensation twice leaves the stock at 5.

Choreography: nobody knows the whole story

In choreography there is no orchestrator. Each service listens to the events that concern it, does its own step and publishes a new one; the compensation is another listening (here Inventory and Orders react to PaymentFailed). The bus in the program is a queue and a map of listeners, with a trace of who received what.

import { Inventory, Payments, Orders } from './services.ts';

type Evt = { type: string; sagaId: string };
const inv = new Inventory(), pay = new Payments(), ord = new Orders();
pay.declines = true; // the payment will be declined

// Minimal bus: a queue and, for each event type, the services that listen to it.
const queue: Evt[] = [];
const listeners = new Map<string, ((e: Evt) => void)[]>();
const trace: string[] = [];
const listen = (type: string, service: string, f: (e: Evt) => void) =>
  listeners.set(type, [...(listeners.get(type) ?? []), (e) => { trace.push(`${e.type.padEnd(20)} -> ${service}`); f(e); }]);
const publish = (type: string, sagaId: string) => queue.push({ type, sagaId });

// Each one knows only its own step and the events it reacts to.
listen('OrderCreated', 'inventory', (e) => {
  try { inv.reserve(`${e.sagaId}:reserve`, 'pen', 2); publish('StockReserved', e.sagaId); }
  catch { publish('StockUnavailable', e.sagaId); }
});
listen('StockReserved', 'payments', (e) => {
  try { pay.pay(`${e.sagaId}:pay`); publish('PaymentConfirmed', e.sagaId); }
  catch { publish('PaymentFailed', e.sagaId); }
});
listen('PaymentConfirmed', 'orders', (e) => ord.setStatus(e.sagaId, 'CONFIRMED'));
listen('PaymentFailed', 'inventory', (e) => inv.release(`${e.sagaId}:release`, 'pen', 2)); // compensates
listen('PaymentFailed', 'orders', (e) => ord.setStatus(e.sagaId, 'CANCELLED'));             // compensates
listen('StockUnavailable', 'orders', (e) => ord.setStatus(e.sagaId, 'CANCELLED'));

ord.setStatus('saga-1', 'PENDING');
publish('OrderCreated', 'saga-1');
for (let e = queue.shift(); e; e = queue.shift()) (listeners.get(e.type) ?? []).forEach((f) => f(e));

console.log(trace.join('\n'));
console.log(`order ${ord.statuses.get('saga-1')} | pen stock ${inv.stock.get('pen')} | receipts ${pay.receipts}`);

// Output:
// OrderCreated         -> inventory
// StockReserved        -> payments
// PaymentFailed        -> inventory
// PaymentFailed        -> orders
// order CANCELLED | pen stock 5 | receipts 0

The trace is the only proof that the saga existed: it says that the order, starting from OrderCreated, went through inventory and payments and that the failure was heard by two services. This readability is fake, because I produced it with a push on the same list: in a real system the trace does not exist, and to know where a saga stands you have to collect the events of several services with a correlation identifier (sagaId).

No service sees the whole path, no single point can stop, and for this very reason an event loop or a forgotten step is discovered late. On a real broker another fact changes too: the listeners are independent consumers, and delivery is at-least-once, so the idempotency key of the previous section is not optional.

A compensation is not a rollback

A rollback makes the effect vanish as if it had never happened; a compensation produces a new one that neutralizes it from the business point of view. The released stock becomes available again, but someone may have seen it sold out in the meantime; a collected payment is refunded, but the customer sees two movements; a confirmation email already sent cannot be recalled, you send another one.

This is why the design work lies in knowing, for each step, what its compensation is and whether it exists. When it does not exist (sending the email), the step must go after those that can fail, not before. The compensation must be repeatable without harm, and must always be able to succeed: if it can fail in turn, the saga needs the BLOCKED state described above.

Orchestration on a managed service: Step Functions

Managed products embed the same two mechanisms. In AWS Step Functions the flow is written as a state machine in Amazon States Language: a Task step can have a Retry block for temporary errors and a Catch block that routes a definitive error to the compensation step. The fragment below is the equivalent schema of the experiment's payment. It was not run (it requires an account and Lambda functions): it is a “schema to adapt”, written from the documentation.

{
  "StartAt": "ProcessPayment",
  "States": {
    "ProcessPayment": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:eu-west-1:111122223333:function:process-payment",
      "Retry": [
        { "ErrorEquals": ["TransientPaymentError"], "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2 }
      ],
      "Catch": [
        { "ErrorEquals": ["PaymentDeclined"], "Next": "ReleaseInventory" }
      ],
      "Next": "ConfirmOrder"
    },
    "ReleaseInventory": { "Type": "Task", "Resource": "arn:aws:lambda:eu-west-1:111122223333:function:release-inventory", "Next": "CancelOrder" },
    "CancelOrder": { "Type": "Task", "Resource": "arn:aws:lambda:eu-west-1:111122223333:function:cancel-order", "Next": "SagaFailed" },
    "SagaFailed": { "Type": "Fail", "Error": "SagaFailed" },
    "ConfirmOrder": { "Type": "Task", "Resource": "arn:aws:lambda:eu-west-1:111122223333:function:confirm-order", "End": true }
  }
}

The Retry reproduces withRetry: the documentation defines IntervalSeconds (wait before the first retry, 1 by default), MaxAttempts (3 by default) and BackoffRate (multiplier of the wait, 2 by default). Here the values are written explicitly so as not to depend on the defaults. The Catch sends PaymentDeclined to ReleaseInventory, then CancelOrder, then a Fail state: the chain of compensations here is written by hand in the graph, not derived automatically. The advantage over code is that the state of every execution is visible in the service; the cost is a lock-in to that platform.

Which one to choose

The practical rule is the number of steps. With two or three services and independent teams, choreography is light and enough; beyond four steps, or when you need to know at every moment where the saga is, orchestration costs less than what its visibility saves. Chris Richardson, who catalogued the pattern, describes both coordinations and points to event sourcing and the transactional outbox as ways to update the state and publish the messages atomically.

This is the weak point that the programs above do not solve: the service that saves and then publishes can die halfway, and it is the topic of the outbox pattern. Retries and compensations that do not succeed are handled with the dead letter queue.

What you can do now

You can explain why a distributed transaction becomes a sequence of steps with compensations, you can implement an orchestrator that compensates in reverse order and retries only temporary errors, and you can recognize the state in which the saga gets stuck. You also know that idempotency keys are the prerequisite of everything, and why the choice between choreography and orchestration is decided on the number of steps and the visibility required. For the basics of how services exchange events, see the comparison between SQS, SNS and EventBridge. If the saga's events end up in an event store, rebuilding the views is covered in Event Sourcing + CQRS; for what the user sees while the saga is half-done, see eventual consistency.