A transaction across services (saga)

Code in:JavaC#PythonJavaScript

In short: A saga is a transaction across several services as a list of steps, each with a compensating step; on failure the compensations run backwards, starting from the last completed step.

Problem

The money is taken, the headphones are gone

A customer orders the last pair of headphones for 59 € (order 1042). The payment goes through. A second later the warehouse answers: "out of stock". Another customer took the last one just before.

var payments = new Payments();
var warehouse = new Warehouse(new() { ["headphones"] = 0 });
var delivery = new Delivery();
var order = new Order(1042, "headphones", 59, "Varna");
try
{
    payments.Charge(order);    // the payment goes through
    warehouse.Reserve(order);  // the warehouse: out of stock
    delivery.Book(order);
}
catch (StepFailed e)
{
    Console.WriteLine($"  ✗ {e.Message}");
}
Console.WriteLine($"charged to the card: {payments.Charged} €");

Output

  ✓ charged 59 €
  ✗ no headphones in stock
charged to the card: 59 €

The program catches the error, but the money stays charged. Tomorrow the customer writes to support, and someone refunds 59 € by hand.

In a monolith with one database the answer is a transaction: all or nothing (you saw it in the lesson on databases and in "ACID: all or nothing"). Here, though, the payment is at an external provider, and the warehouse is a separate service with its own database. There is no transaction that covers all three.

The idea

A saga: steps with reverse steps

If you cannot go back, you take new actions that repair the consequences. Each step of the order comes with a reverse step (a compensation):

Step	Service	Reverse step
charges 59 €	payments	refunds 59 €
reserves the headphones	warehouse	releases them
books a courier	delivery	cancels the courier

Each step is a small local transaction in its own service. If a step refuses, the reverse steps of the steps already done run backwards: the last step done is the first undone. This is called a saga.

A reverse step is not a rollback. A rollback erases, as if nothing had happened. A refund of 59 € is a second payment in the other direction, which the customer sees in the bank next to the first. So between the steps the system is in an intermediate state ("the order is in progress"), and everyone must cope with it.

Here the saga is led by one object, the orchestrator: it knows the steps, calls them in order, and on a failure calls the compensations. The steps are in a list and the compensations on a stack: the last one in is the first out.

See it

A saga in code

The refusal of a service is an exception. In this small system the services are objects in one process, and the refusals follow a script: there is one pair of headphones, and there is no courier to Vidin.

record Order(int Id, string Item, int Price, string City);

class StepFailed(string reason) : Exception(reason);

record Step(Action<Order> Do, Action<Order> Undo);

class Warehouse(Dictionary<string, int> stock)
{
    public void Reserve(Order o)
    {
        int left = stock.GetValueOrDefault(o.Item);
        if (left == 0) throw new StepFailed($"no {o.Item} in stock");
        stock[o.Item] = left - 1;
        Console.WriteLine($"  ✓ reserved: {o.Item}");
    }

    public void Release(Order o)
    {
        stock[o.Item] = stock.GetValueOrDefault(o.Item) + 1;
        Console.WriteLine($"  ↩ released: {o.Item}");
    }
}

Step is a record with two Action<Order> delegates: one does the step, the other undoes it. Payments and delivery are like the warehouse: one method for the step and one for the reverse.

The orchestrator:

class Saga(List<Step> steps)
{
    public string Run(Order order)
    {
        var undo = new Stack<Action<Order>>();
        try
        {
            foreach (var step in steps)
            {
                step.Do(order);
                undo.Push(step.Undo);
            }
            return "confirmed";
        }
        catch (StepFailed e)
        {
            Console.WriteLine($"  ✗ {e.Message}");
            while (undo.Count > 0)
                undo.Pop()(order);
            return "cancelled";
        }
    }
}

Step by step, in words

1. Order 1043: a charger for 25 € to Vidin. Each step lives in another service, and each one has a reverse: charge ↔ refund, reserve ↔ release, book ↔ cancel.
2. Step 1: Payments charges 25 €. Its reverse step goes on the undo stack (Push).
3. Step 2: Warehouse reserves the charger. "Release" goes on top of "refund".
4. Step 3: no courier to Vidin, and Delivery refuses. The first two steps have already happened in other systems, and there is no rollback across them.
5. The saga catches the refusal and turns back: it takes the compensations off the stack, top first.
6. Pop gives the last one in: the charger is released first. The last step done is the first undone.
7. Then the payment: 25 € go back to the card. A refund is a new action, not an erased one: the customer sees both in the bank.
8. Cancelled: the charger is back in stock and 0 € is kept. Every system agrees again, through later actions instead of one transaction.

Three orders: 1041 goes through, 1042 stops at the warehouse, 1043 stops at delivery.

var payments = new Payments();
var warehouse = new Warehouse(new() { ["headphones"] = 1, ["charger"] = 5 });
var delivery = new Delivery();
var saga = new Saga([
    new Step(payments.Charge, payments.Refund),
    new Step(warehouse.Reserve, warehouse.Release),
    new Step(delivery.Book, delivery.Cancel),
]);
Order[] orders = [
    new Order(1041, "headphones", 59, "Sofia"),
    new Order(1042, "headphones", 59, "Varna"),
    new Order(1043, "charger", 25, "Vidin"),
];
foreach (var order in orders)
{
    Console.WriteLine($"order {order.Id}");
    Console.WriteLine($"  = {saga.Run(order)}");
}
Console.WriteLine($"charged to cards: {payments.Charged} €");

Output

order 1041
  ✓ charged 59 €
  ✓ reserved: headphones
  ✓ courier to Sofia
  = confirmed
order 1042
  ✓ charged 59 €
  ✗ no headphones in stock
  ↩ refunded 59 €
  = cancelled
order 1043
  ✓ charged 25 €
  ✓ reserved: charger
  ✗ no courier to Vidin
  ↩ released: charger
  ↩ refunded 25 €
  = cancelled
charged to cards: 59 €

In the end 59 € are charged to cards: only for the order that went through.

Try it

Predict the output

The same saga, but the warehouse is the first step and the payment the second. The headphones are sold out. What will it print?

var payments = new Payments();
var warehouse = new Warehouse(new() { ["headphones"] = 0 });
var delivery = new Delivery();
var saga = new Saga([
    new Step(warehouse.Reserve, warehouse.Release),
    new Step(payments.Charge, payments.Refund),
    new Step(delivery.Book, delivery.Cancel),
]);
Console.WriteLine($"  = {saga.Run(new Order(1042, "headphones", 59, "Varna"))}");

Answer

  ✗ no headphones in stock
  ↩ released: headphones
  = cancelled

  ✗ no headphones in stock
  = cancelled

  ✗ no headphones in stock
  ↩ refunded 59 €
  = cancelled

Check

The idea

What cannot be undone

Not every step has a reverse. The email "your order is confirmed" cannot be sent back. A parcel the courier has already picked up does not come back with one call.

So such steps go last, after everything that can refuse. The step after which the saga only moves forward is called the pivot. After it, failures are not compensated but retried until they succeed: with timeout and retry.

Try it

Find the bug

A colleague reorders the loop, and the tests pass: on a failure the money is refunded. A week later accounting finds refunds for payments that never happened: the card was declined, and the customer got 59 €. Which line is the bug on?

Pick the line with the bug.

1class Saga(List<Step> steps)2{3 public string Run(Order order)4 {5 var undo = new Stack<Action<Order>>();6 try7 {8 foreach (var step in steps)9 {10 undo.Push(step.Undo);11 step.Do(order);12 }13 return "confirmed";14 }15 catch (StepFailed e)16 {17 Console.WriteLine($" ✗ {e.Message}");18 while (undo.Count > 0)19 undo.Pop()(order);20 return "cancelled";21 }22 }23}

Check

The idea

Orchestration or choreography

The orchestrator is one place that knows the whole process. The other way has no such place: each service listens to events and answers with its own. Payments hear "order" and say "charged". The warehouse hears "charged" and says "reserved" or "out of stock". Payments hear "out of stock" and refund the money. This is called choreography.

The bus is a queue in memory with subscriptions, as in the lesson on events:

record Event(string Type, int Order);

class Bus
{
    readonly Dictionary<string, List<Action<Event>>> handlers = [];
    readonly Queue<Event> queue = new();

    public void On(string type, Action<Event> handler)
    {
        if (!handlers.TryGetValue(type, out var list))
            handlers[type] = list = [];
        list.Add(handler);
    }

    public void Publish(Event e) => queue.Enqueue(e);

    public void Run()
    {
        while (queue.TryDequeue(out var e))
        {
            Console.WriteLine($"{e.Type} {e.Order}");
            foreach (var handler in handlers.GetValueOrDefault(e.Type, []))
                handler(e);
        }
    }
}

The same refusal of 1042, as a chain of events:

var bus = new Bus();
// payments
bus.On("OrderPlaced", e => bus.Publish(new Event("PaymentCharged", e.Order)));
bus.On("StockMissing", e => bus.Publish(new Event("PaymentRefunded", e.Order)));
// the warehouse (in this example 1042 is out of stock)
bus.On("PaymentCharged", e => bus.Publish(new Event("StockMissing", e.Order)));
// orders
bus.On("PaymentRefunded", e => bus.Publish(new Event("OrderCancelled", e.Order)));
bus.Publish(new Event("OrderPlaced", 1042));
bus.Run();

Output

OrderPlaced 1042
PaymentCharged 1042
StockMissing 1042
PaymentRefunded 1042
OrderCancelled 1042

	Orchestration	Choreography
Where the process is	in one object	spread across subscriptions
A new step	a line in the list	a new subscription and a new event
"Why did 1042 stop?"	the state of the orchestrator	you gather the events from every service
Coupling	the orchestrator knows everyone	each service knows only the events

For two or three steps choreography is lighter. When there are five steps with conditions, the process gets lost, and the orchestrator is the clearer choice.

The idea

The compensation arrives twice

Between services, commands and events go through a queue, and the queue delivers at least once (you saw it in the lesson on events). If "refund 59 € for 1042" arrives twice, the customer gets 118 €.

So every step and every compensation is idempotent: a second time with the same input does nothing. The key is the action and the order. Towards the real provider it is sent as an idempotency key, so that the provider refuses the repeated refund too.

class Refunds
{
    readonly HashSet<string> done = [];
    public int Total { get; private set; }

    public void Refund(int order, int euros)
    {
        // the key of the step: action + order
        if (!done.Add($"refund:{order}"))
        {
            Console.WriteLine($"  · {order}: already refunded");
            return;
        }
        Total += euros;
        Console.WriteLine($"  ↩ refunded {euros} € for {order}");
    }
}

var refunds = new Refunds();
// the queue delivers "at least once": 1042 arrives twice
refunds.Refund(1042, 59);
refunds.Refund(1042, 59);
refunds.Refund(1043, 25);
Console.WriteLine($"refunded in total: {refunds.Total} €");

Output

  ↩ refunded 59 € for 1042
  · 1042: already refunded
  ↩ refunded 25 € for 1043
refunded in total: 84 €

Try it

Fill in

What checks whether the refund has already been made?

if (!done.($"refund:{order}"))
{
    Console.WriteLine($"  · {order}: already refunded");
    return;
}

Check

The idea

When a saga hurts

A saga is not free:

• The code doubles. Every step needs a reverse, tests for both, and tests for a failure at every point.
• Intermediate states are visible. The customer sees "in progress", and the money is charged for a moment before it comes back. Someone else may read an order that is then cancelled: a saga has no isolation like a transaction.
• A failure is harder to investigate. The steps are in different services, with different logs.

So do not use it where a transaction is enough. If stock and orders are modules of one monolith with one database, a transaction keeps them safe more simply and more surely. A saga is for the steps that leave the system: an external provider, another service, another company.

More precisely, if you are curious

Why not two phases (2PC)? There is a protocol for one transaction across several databases: the coordinator asks everyone "are you ready?" and then tells everyone "commit". The external payment provider does not take part in such a protocol. And while the coordinator waits, everyone holds locked rows; if it crashes halfway, they stay locked. That is why services almost always use a saga.

The orchestrator crashes too. If the process stops after the payment, how does it know how far it got? A real orchestrator stores the state of each saga in a database (step 2 of 3, order 1042) and continues after a restart. The commands to the services go out through an outbox, so they are not lost between the write and the send.

A semantic lock. An order "in progress" is a sign to others: do not invoice it, do not allow an address change. This makes up for the missing isolation.

Ready-made tools: Temporal, Camunda and AWS Step Functions keep the state of long processes and retry the steps. There are also saga libraries in the frameworks: Axon (Java), MassTransit and NServiceBus (.NET). The queue between services in a real project is Kafka or RabbitMQ. How they are run and monitored is in the DevOps course ZeroToOps.

The name comes from a 1987 paper by Hector Garcia-Molina and Kenneth Salem on long-lived transactions in databases.

Exercise

Loyalty points

Solve it in your editor.

The shop gives 1 point for every full 10 €. Add a service Loyalty with a method for the step (gives the points) and one for the reverse (takes them away). Put it in the saga as the third step, between the warehouse and delivery. Run order 1044: a speaker for 120 € to Vidin, with 2 speakers in stock. At the end print the points and the money charged.

Output

order 1044
  ✓ charged 120 €
  ✓ reserved: speaker
  ✓ +12 points
  ✗ no courier to Vidin
  ↩ -12 points
  ↩ released: speaker
  ↩ refunded 120 €
  = cancelled
points: 0, charged: 0 €

More tasks

• A declined card. Make the payment refuse above 100 € ("card declined") and run 1044 to Sofia. What is on the stack when the payment refuses?
• Choreography. Add the events StockReserved and OrderConfirmed to the bus and run an order that is in stock.

Test

Check yourself

1. Why can the order not be kept safe with one database transaction, as in the monolith?

Transactions are slow, so shops do not use themIt can, with a higher isolation levelThe payment is at an external provider and the warehouse is another service with its own database: there is no shared transaction that covers all three

2. The steps are payment, warehouse, delivery. Delivery refuses. In what order do the compensations run?

First the money is refunded, then the goods are releasedDelivery is cancelled too, then the warehouse, then the paymentFirst the goods are released, then the money is refunded: the last step done is the first undone

3. Why must the compensation "refund 59 € for 1042" be idempotent?

The queue delivers messages at least once, so the same compensation can arrive twice, and the customer must get the money onceSo that it can be undone tooTo make it faster

4. When is choreography (services react to events) the better choice than orchestration?

When you want to see the whole process in one placeWhen there are two or three steps that rarely change: there is no central object, and each service knows only its own eventsAlways, because there is no central point

5. The shop is a monolith with one database: orders, stock and payments in different tables. The payment is at an external provider. Do you need a saga for the stock and the order?

Yes, because the tables are differentNo: the stock and the order are in one database and one transaction keeps them safe. The saga is only for the steps that leave the systemYes: every order must be a saga

Check

Start the lesson over

Related lessons

• Builds on: Transactional outbox · API design: contract, versions, retried requests · Trade-offs · When the neighbour is slow: timeout, retry, circuit breaker
• Continues in: Integrating old and foreign systems · Changing a live system (strangler fig, expand–contract) · Your first architecture task
• Tasks on this topic: The architect at work