Transactional outbox pattern in JavaScript

Problem

Paid, and no parcel leaves

Since the previous lesson the shop publishes an event as soon as the order is saved. The save goes to the database, and the event goes to the broker. Here are two ways to write that. In them db is the database, queue is the broker's queue, and crash.at is a place where the process may die according to a recorded script:

// NOT LIKE THIS: the save and the event are in two separate systems
function saveThenPublish(id, total) {
  db.transaction((tx) => tx.orders.set(id, total));
  crash.at("after the save");
  queue.push(`evt-${id}: OrderPlaced ${id}`);
}

function publishThenSave(id, total) {
  queue.push(`evt-${id}: OrderPlaced ${id}`);
  db.transaction((tx) => {
    tx.orders.set(id, total);
    crash.at("before commit");
  });
}

In each function the process dies once, at its own place:

crash = new CrashScript("after the save");
try {
  saveThenPublish(1001, 120);
} catch (e) {
  if (!(e instanceof Crash)) throw e;
  console.log(`✗ the process dies ${e.message}`);
}
crash = new CrashScript("before commit");
try {
  publishThenSave(1002, 80);
} catch (e) {
  if (!(e instanceof Crash)) throw e;
  console.log(`✗ the process dies ${e.message}`);
}
console.log(`orders: [${[...db.orders.keys()].join(", ")}]`);
console.log(`queue: [${queue.join(", ")}]`);

✗ the process dies after the save
✗ the process dies before commit
orders: [1001]
queue: [evt-1002: OrderPlaced 1002]

Order 1001 is in the database, but there is no event for it. The customer has paid, and the warehouse does not know it has to send a parcel. The event for 1002 is in the queue, but order 1002 is not there: the transaction rolled back. The warehouse packs an order that does not exist.

A crash is not rare. The process restarts on every deploy, the machine loses power, the network to the broker drops for a second. With thousands of orders a day, some order lands exactly between the two lines.

The idea

Two systems, no shared transaction

A transaction promises "all or nothing", but only for the writes in its own database. The broker is another system. The database commit knows nothing about it, and a message sent to the broker cannot be "taken back". So whatever the order of the two lines, a crash between them leaves one system without the other.

There is a protocol for a shared transaction across two systems (two-phase commit), but most brokers do not support it, and where it exists, it is slow and fragile.

The fix is to not have two systems at the moment of the save. The event is written as a row in a table of the same database, in the same transaction as the order. The table is called the outbox. Commit writes both rows, or neither. Then a separate relay reads the outbox, sends the rows not sent yet to the broker, and marks them. This is the transactional outbox pattern.

See it

The database and the crash in memory

The database here is an object with two tables. The transaction collects the writes in Tx and puts them in the tables at once, only if the work gets to the end. If it throws halfway, nothing is written: that is a rollback.

// the writes of one transaction, before commit
class Tx {
  orders = new Map();
  outbox = [];
}

class Db {
  orders = new Map();
  outbox = []; // { id, event, sent }

  // all or nothing: the writes reach the tables only if the work gets to the end
  transaction(work) {
    const tx = new Tx();
    work(tx);
    for (const [id, total] of tx.orders) this.orders.set(id, total); // commit
    this.outbox.push(...tx.outbox);
  }
}

The work is a lambda that gets tx. If it throws, the lines after its call (the commit) do not run.

The crash is an exception at an exact place. The database and the queue are separate systems: they survive the crash, and the program goes on like a restarted process.

class Crash extends Error {}

// a recorded script: where the process dies (no point: nowhere)
class CrashScript {
  constructor(point = "") {
    this.point = point;
  }

  at(here) {
    if (here === this.point) throw new Crash(here);
  }
}

const db = new Db();
const queue = []; // the broker's queue
let crash = new CrashScript();

You saw how the same looks with a real database in the databases lesson.

See it

The order and the outbox row together

Saving the order also puts a row in the outbox, in the same transaction. It sends nothing to the broker:

function placeOrder(id, total) {
  db.transaction((tx) => {
    tx.orders.set(id, total);
    // in the same transaction
    tx.outbox.push({ id: `evt-${id}`, event: `OrderPlaced ${id}`, sent: false });
    crash.at("before commit");
  });
}

The relay goes through the outbox, sends every row not sent yet, and marks it:

// the relay sends the unsent rows of the outbox
function relay() {
  for (const row of db.outbox) {
    if (row.sent) continue;
    queue.push(`${row.id}: ${row.event}`);
    crash.at("in the relay");
    row.sent = true;
  }
}

function show() {
  console.log(`orders: [${[...db.orders.keys()].join(", ")}]`);
  console.log(`outbox: [${db.outbox.map((r) => r.id + (r.sent ? " ✓" : "")).join(", ")}]`);
  console.log(`queue: [${queue.join(", ")}]`);
}

Order 1003 goes through. Order 1004 dies before commit. Then the relay starts, as after a restart:

placeOrder(1003, 120);
show();
crash = new CrashScript("before commit");
try {
  placeOrder(1004, 80);
} catch (e) {
  if (!(e instanceof Crash)) throw e;
  console.log(`✗ the process dies ${e.message}`);
}
crash = new CrashScript();
relay(); // after the restart
show();

orders: [1003]
outbox: [evt-1003]
queue: []
✗ the process dies before commit
orders: [1003]
outbox: [evt-1003 ✓]
queue: [evt-1003: OrderPlaced 1003]

function placeOrder(id, total) {
  db.transaction((tx) => {
    tx.orders.set(id, total);
    // in the same transaction
    tx.outbox.push({ id: `evt-${id}`, event: `OrderPlaced ${id}`, sent: false });
    crash.at("before commit");
  });
}

// NOT LIKE THIS: the save and the event are in two separate systems
function saveThenPublish(id, total) {
  db.transaction((tx) => tx.orders.set(id, total));
  crash.at("after the save");
  queue.push(`evt-${id}: OrderPlaced ${id}`);
}

function placeOrder(id, total) {
  db.transaction((tx) => {
    tx.orders.set(id, total);
    // in the same transaction
    tx.outbox.push({ id: `evt-${id}`, event: `OrderPlaced ${id}`, sent: false });
    crash.at("before commit");
  });
}

// the relay sends the unsent rows of the outbox
function relay() {
  for (const row of db.outbox) {
    if (row.sent) continue;
    queue.push(`${row.id}: ${row.event}`);
    crash.at("in the relay");
    row.sent = true;
  }
}

Step by step, in words

Scenario 1: save first, then the event. Order 1001 goes into the database with a commit.
The process dies right between the two lines. The order is in the database, but the event never leaves.
This line never runs. The warehouse never hears of 1001: the customer paid and no parcel will leave.
Scenario 2: the other way round. The event evt-1002 leaves first.
Then the order is written in a transaction.
The process dies before the commit, and the database drops the write. Order 1002 does not exist.
But the event is already in the queue: the warehouse packs an order that does not exist.
Scenario 3, the outbox: order 1003 and its event, as a row in the outbox table, in one transaction.
Commit: both rows go in at once. Either both or neither.
The process dies before it sends anything. Nothing is lost: the row waits in the outbox.
After the restart the relay reads the outbox and finds the row that is not sent.
It sends evt-1003 to the queue, and the warehouse gets it.
Only then does it tick the row off. A crash between the two sends the row again, so the consumer must be idempotent.
The save and the event no longer drift apart: both arrive, sooner or later.

After commit the order and the outbox row are in the database, and the queue is still empty: the process may die here and nothing is lost. Order 1004 leaves nothing behind: no order and no row, because both are in one transaction. The relay finds the row not sent yet and sends it.

In a real system the relay is a loop that asks the database every second: "give me the rows not sent yet". That is why the event leaves a little after the commit, not at the same instant.

Try it

Predict the output

The relay sends the row and dies before marking it. After the restart it runs again. What is in the queue?

The last line of the output is:

placeOrder(1003, 120);
crash = new CrashScript("in the relay");
try {
  relay();
} catch (e) {
  if (!(e instanceof Crash)) throw e;
  console.log(`✗ the process dies ${e.message}`);
}
crash = new CrashScript();
relay(); // after the restart
show();

Answer

queue: [evt-1003: OrderPlaced 1003]

queue: [evt-1003: OrderPlaced 1003, evt-1003: OrderPlaced 1003]

queue: []

Try it

Find the bug

A colleague decides that duplicates are a nuisance and swaps two lines in the relay. The crash is at the same place, between the two lines:

✗ the process dies in the relay
orders: [1003]
outbox: [evt-1003 ✓]
queue: []

The outbox row is marked, and the queue is empty. Which line has the bug?

Pick the line with the bug.

Try it

Put the lines in order

Put the body of the relay loop in an order where nothing is lost.

row.sent = true;
if (row.sent) continue;
queue.push(`${row.id}: ${row.event}`);

The idea

When an outbox is not worth it

An outbox adds a table, a relay and a delay. Do not use it when:

there is no separate broker. If the reactions are in the same process and write to the same database (Observer in the same transaction), one transaction is enough;
a lost message does no harm. Updating statistics or warming a cache can be lost once a month without anyone getting hurt;
the event must leave in the same millisecond. The relay reads at intervals. A second of delay is usually fine, but not always.

And when you do use it, think about:

duplicates. The relay sends at least once. Consumers must be idempotent;
order. One relay sends in outbox order. Several relays at once can swap the order or send one row twice, so rows are locked when read;
growth. Sent rows are deleted after a few days, otherwise the table grows forever.

On the other side there is a mirror pattern, the inbox: the consumer writes the id of the received event in its own database, in the same transaction as its work. That is the warehouse's seen from the previous lesson, but in a table.

More precisely, if you are curious

The relay in a real system. The simplest one is a loop with the query SELECT … FROM outbox WHERE sent = false ORDER BY id LIMIT 100. In PostgreSQL FOR UPDATE SKIP LOCKED lets several relays run without taking the same row. Another way is change data capture: Debezium reads the database log (the WAL in PostgreSQL, the binlog in MySQL) and sends new outbox rows to Kafka, with no queries at intervals. How Debezium or Kafka Connect is run and monitored is a topic for the DevOps course ZeroToOps.

Ready-made solutions. Spring Modulith has an event publication registry, which is an outbox. In .NET MassTransit and NServiceBus have a built-in outbox on top of Entity Framework. In Python and Node.js it is usually written by hand, with a table and a small relay, as in this lesson.

Two-phase commit. XA transactions (JTA in Java, System.Transactions in .NET) promise a shared transaction across a database and a broker. In practice Kafka does not support them, and neither do RabbitMQ or the cloud queues. That is why the outbox is the usual answer.

Why not just retry? "If the send fails, try again" helps with an error from the broker, but not with a crash of the process: after the restart nobody remembers that something had to leave. The outbox is the memory that survives the crash.

Exercise

A relay in batches

Solve this in your own editor.

In a real system the relay reads at most a few rows at once (LIMIT in the query). Write relay(limit), which sends at most limit rows not sent yet and returns how many it sent. Save three orders and run the relay with limit 2 until it returns 0:

round 1: sent 2
round 2: sent 1
round 3: sent 0
queue: 3

More tasks

A cancelled order. Add cancelOrder(id), which in one transaction removes the order and puts OrderCancelled in the outbox. Check that a crash before commit does not remove the order and sends nothing.
Cleanup. Add the moment of sending (a tick) to the row, and a function that deletes sent rows older than 10 ticks.
Inbox. Write a warehouse that keeps the ids of the received events and drops the duplicate from the "Predict the output" card.

Test

Check yourself

1. Why does the database transaction not protect sending to the broker too?It does, if the send is inside the transaction blockThe broker is another system: the database commit knows nothing about it, and the other way roundBecause transactions are read-only

2. What does the shop write in one transaction when it uses an outbox?Only the event, and the relay writes the orderThe order and a row in the outbox table with the eventThe order, and it sends the event right after commit

3. The relay sends a row and dies before marking it. What happens after the restart?The event is lostNothing: the broker remembers it got itIt sends the same event again: the queue has a duplicate and the consumer must be idempotent

4. When is an outbox unnecessary?Always, when there is a databaseWhen there are more than a hundred orders a dayWhen there is no separate broker: the subscribers are in the same process and work in the same transaction

Transactional outbox

Cookies