Strangler fig and expand–contract: zero-downtime migration

Problem

The big switch-over

The orders module is from 2016. It computes the total: under 50 € the customer pays 5 € delivery, from 50 € up the delivery is free. The team writes a new module, with new code and a new table. The plan: on Saturday night the site is stopped, the old one is switched off, the new one is switched on.

Think about what happens if the new one has a bug that the tests did not catch. On Monday customers in three countries see wrong totals. Going back to the old one is hard by now: since Saturday the new one writes orders into the new table, and the old one does not know them. The choice is between bad and worse: fix it live, or go back and lose orders.

The problem is not the new code. The problem is that the switch-over is one big step with no way back.

The idea

Strangler fig: the new one grows around the old

The name is a plant's: the strangler fig sprouts on another tree, sends roots down around it and over the years replaces it. The tree disappears, and the shape stays.

A system is replaced the same way. A go-between (here the Router) is put in front of the old one, and the rest of the shop talks to it. At first it sends everything to the old one. Then the new one takes a small part, then a bigger one, finally everything. The old one is switched off last, when it no longer gets anything. This is a strangler fig.

Each step is small and has a way back: if the new one misbehaves, the percent is lowered. The Router is the same kind of place as an anti-corruption layer: a boundary behind which whatever is there can be replaced without the others noticing.

See it

The Router in code

Both modules answer the same question: how much the order costs.

record Order(int Id, int Subtotal);

interface IOrders
{
    int Total(Order order);
}

// the old module (2016)
class Legacy : IOrders
{
    public int Total(Order order) =>
        order.Subtotal >= 50 ? order.Subtotal : order.Subtotal + 5;
}

static class Shop
{
    public const int FreeFrom = 50, Delivery = 5;
}

class Fresh : IOrders
{
    public int Total(Order order)
    {
        var free = order.Subtotal >= Shop.FreeFrom;
        return order.Subtotal + (free ? 0 : Shop.Delivery);
    }
}

Legacy and Fresh implement one interface IOrders, so the shop does not know which of the two answers it. The Router is such a module too:

class Router(IOrders legacy, IOrders fresh, int percent)
    : IOrders
{
    public int ByLegacy { get; private set; }
    public int ByFresh { get; private set; }
    public List<int> Mismatches { get; } = [];

    public int Total(Order order)
    {
        // the same order always goes to the same place
        if (order.Id % 10 * 10 < percent)
        {
            ByFresh++;
            return fresh.Total(order);
        }
        ByLegacy++;
        var answer = legacy.Total(order);
        // shadow: the new one computes, never answers
        if (fresh.Total(order) != answer)
            Mismatches.Add(order.Id);
        return answer;
    }
}

The order goes to the new one if its last digit times 10 is below the percent: at 10% only the digit 0, at 50% the digits 0 to 4. When the old one answers, the new one computes the same order in the shadow: its result is only compared, the customer does not see it.

See it

In the shadow

The first version of the new one:

// the first version of the new one: free delivery forgotten
class FreshDraft : IOrders
{
    public int Total(Order order) => order.Subtotal + Shop.Delivery;
}

Ten orders, at 0%:

Order[] orders = [
    new(1001, 30), new(1002, 64), new(1003, 18), new(1004, 45), new(1005, 120),
    new(1006, 9), new(1007, 50), new(1008, 22), new(1009, 75), new(1010, 40),
];
var router = new Router(new Legacy(), new FreshDraft(), 0);
foreach (var o in orders) router.Total(o);
Console.WriteLine($"0%: old {router.ByLegacy}, new {router.ByFresh}");
Console.WriteLine($"mismatches: [{string.Join(", ", router.Mismatches)}]");

0%: old 10, new 0
mismatches: [1002, 1005, 1007, 1009]

All 10 are answered by the old one, and all totals are right. And the list says that the new one differs on 4 orders, all of them from 50 € up: it forgot the free delivery. The bug is found on the real orders, without a single customer seeing it. The fix is Fresh from above.

class Router(IOrders legacy, IOrders fresh, int percent)
    : IOrders
{
    public int ByLegacy { get; private set; }
    public int ByFresh { get; private set; }
    public List<int> Mismatches { get; } = [];

    public int Total(Order order)
    {
        // the same order always goes to the same place
        if (order.Id % 10 * 10 < percent)
        {
            ByFresh++;
            return fresh.Total(order);
        }
        ByLegacy++;
        var answer = legacy.Total(order);
        // shadow: the new one computes, never answers
        if (fresh.Total(order) != answer)
            Mismatches.Add(order.Id);
        return answer;
    }
}

Step by step, in words

Router stands in front of the old orders module (2016) and the new one. The shop talks only to Router; it decides who answers.
0%: the old one answers all 10. The new one computes in the shadow and differs in 4: 1002, 1005, 1007, 1009. All are over 50 €: it forgot the free delivery.
Fixed and still at 0%: 0 mismatches. Customers have not seen a single wrong total.
10%: the order whose last digit is 0 (1010) goes to the new one. The same order always goes the same way, so a customer does not jump between the two.
50%: 5 of 10 to the new one. If it misbehaves, the way back is one number: 10 again, and the same orders return to the old one.
100%: the new one answers everything. The old one gets no traffic, but it still runs and still has its data.
Switching the old one off is the point of no return. It comes last, after weeks with 0 mismatches, and with its data archived.

See it

Step by step

With the fixed new one, through the whole plan:

Order[] orders = [
    new(1001, 30), new(1002, 64), new(1003, 18), new(1004, 45), new(1005, 120),
    new(1006, 9), new(1007, 50), new(1008, 22), new(1009, 75), new(1010, 40),
];
var legacy = new Legacy();
var fresh = new Fresh();
foreach (var percent in new[] { 0, 10, 50, 100 })
{
    var router = new Router(legacy, fresh, percent);
    foreach (var o in orders) router.Total(o);
    Console.WriteLine($"{percent,3}%: new {router.ByFresh}, old {router.ByLegacy}, mismatches {router.Mismatches.Count}");
}

  0%: new 0, old 10, mismatches 0
 10%: new 1, old 9, mismatches 0
 50%: new 5, old 5, mismatches 0
100%: new 10, old 0, mismatches 0

The mismatches stay 0 at 10% and at 50% too. In a real shop days pass between two steps: each step is watched until the errors, the response time and the complaints show that it is calm.

The choice is by the number of the order, not random. So an order always goes to the same place, and if the percent is lowered from 50 to 10, the same orders go back to the old one. In a real project the choice is by something that keeps the customer in one place: the number of the customer or of the basket.

The percent is a setting (a flag), not code: in a real project it changes without a new release, so going back takes seconds too.

Try it

Predict the output

int percent = 30;
foreach (var id in new[] { 1007, 1012, 1020 })
    Console.WriteLine($"{id} {(id % 10 * 10 < percent ? "new" : "old")}");

Answer

1007 old
1012 old
1020 old

1007 old
1012 new
1020 new

1007 new
1012 old
1020 old

Try it

Find the bug

A colleague writes a shorter Router: "10% is 10%, let it be random". Customers start writing that the total in the basket changes on every reload. Which line is the bug on?

Pick the line with the bug.

The idea

The data: expand–contract

The code is replaced with the Router. The data is harder: the new one wants a new table. In the old one the total is text with a decimal comma ("64,00"), in the new one it is a number in cents (6400). And while the traffic moves, both tables have to be right.

In the lesson on migrations changing a column was done in steps: expand, transition, contract. Here it is the same, but for a whole table and with a check:

Expand: the new table is added next to the old one. Nothing is removed.
Dual write: every new order is written into both.
Backfill: the history (the orders from before the dual write) is copied into the new one.
Reconciliation: the two are compared until the mismatches are 0.
Reads move to the new one, the old one is no longer written.
Contract: the old table is removed.

This is expand–contract. Up to step 6 each step can be undone: the old table is still there and it is right.

See it

Dual write, backfill and reconciliation

The two tables here are SortedDictionarys (sorted by number), not a database, so that the logic can be seen:

class Migration
{
    // the old table (2016): the total is text with a decimal comma
    public SortedDictionary<int, string> OldRows { get; } = [];
    // the new table: the total is in cents
    public SortedDictionary<int, long> NewRows { get; } = [];

    static long Cents(string text)
    {
        var p = text.Split(',');
        return long.Parse(p[0]) * 100 + long.Parse(p[1]);
    }

    // dual write: every new order goes into both
    public void Place(int id, string total)
    {
        OldRows[id] = total;
        NewRows[id] = Cents(total);
    }

    // backfill: copies the history without writing over new rows
    public void Backfill()
    {
        foreach (var (id, total) in OldRows)
        {
            try
            {
                NewRows.TryAdd(id, Cents(total));
            }
            catch (FormatException)
            {
                Console.WriteLine($"✗ backfill {id}: \"{total}\"");
            }
        }
    }

    // reconciliation: every old row must be in the new table too
    public List<int> Reconcile() =>
        OldRows.Keys.Where(id => !NewRows.ContainsKey(id)).ToList();
}

var m = new Migration();
// the history: orders from before the migration, only in the old table
m.OldRows[1001] = "35,00";
m.OldRows[1002] = "64,00";
m.OldRows[1003] = "23,00";
m.OldRows[1004] = "5O,00";
// from today on: dual write
m.Place(1011, "42,50");
m.Place(1012, "99,90");
m.Backfill();
var missing = m.Reconcile();
Console.WriteLine($"old: {m.OldRows.Count}, new: {m.NewRows.Count}, missing in the new one: [{string.Join(", ", missing)}]");
Console.WriteLine(missing.Count == 0 ? "reads can move to the new table" : "not yet: fix the difference first");

✗ backfill 1004: "5O,00"
old: 6, new: 5, missing in the new one: [1004]
not yet: fix the difference first

Order 1004 from 2016 has the letter O instead of a zero in its total. The old module lives with that, but turning it into cents cannot. Backfill does not hide it: it says it out loud and goes on. Reconciliation catches it and says "not yet". Someone will fix the row in the old table, backfill will run again, and reconciliation will show 0.

Backfill can be run as many times as needed: it copies only what is missing. The dual write can be done as two writes in one transaction (the lesson on databases) or through an outbox, so that no write ends up in only one of the tables.

Try it

Fill in the blank

Backfill must not write over a row that is already in the new table. How?

foreach (var (id, total) in OldRows)
    NewRows.(id, Cents(total));

Try it

Put the steps in order

StopOldWrites(); // the old one read-only
Reconcile(); // compare until the mismatches are 0
SwitchReads(); // reads from the new one
Expand(); // the new table next to the old one
Backfill(); // the history into the new one
Contract(); // drop the old one
DualWrite(); // every write into both

The idea

The point of no return

Every step so far has a way back: the percent is lowered, reads go back to the old table, the dual write goes on. Two steps do not: switching off the old module and removing the old table. After them, going back means restoring from an archive.

So they come last and are decided on purpose. Before them: weeks with 0 mismatches in reconciliation, no code that reads or writes the old one, and an archive of the data. It is good to write down the plan, the criterion for each step and how it is undone in an ADR before starting: halfway through a migration everyone wants to finish it faster.

A migration stopped halfway is the worst: two systems, two sets of data and a dual write, all kept alive forever. The plan has to reach contract.

The idea

When it hurts

A small system with little data. If stopping for half an hour at night is acceptable and the data moves in minutes, a Router, a shadow and reconciliation are weeks of work with no gain. One migration with a plan for going back (an archive before it) is enough.
When there is nowhere to put a Router. If the old one is called directly from twenty places, first there has to be one boundary in front of it (a facade, a hub), and only then can it be strangled.
The shadow is not free. The new one computes everything once more. If it sends emails or takes money, that must not happen in the shadow: the shadow has to be free of side effects.
An endless "almost". If 100% never comes (the last 5% are the hardest), two systems remain to maintain. Without a deadline and a criterion, a strangler fig turns into two monoliths instead of one.

More precisely, if you are curious

The name comes from an article by Martin Fowler from 2004 (Strangler Fig Application). Expand–contract is also described as parallel change.

The Router in a real project is most often in a gateway or load balancer in front of the two systems: a rule by path (/orders to the new one) or by percent. They are deployed and monitored as infrastructure, and that is in the DevOps course ZeroToOps. The percent is often a feature flag: a setting that changes without a new release (LaunchDarkly, Unleash, OpenFeature).

Schema migrations are written with tools like Flyway and Liquibase (Java), EF Core Migrations (.NET), Alembic (Python), Prisma or Knex (JavaScript). They run the steps in order, but they do not decide when: that is the expand–contract plan.

Instead of a dual write in the code CDC (change data capture) is sometimes used: a tool like Debezium reads the log of the old database and carries each change into the new one. The code is not touched, but one more system is added to watch.

The shadow comparison at large scale is done asynchronously: the answers are recorded and compared later, so that the new one does not slow the customer down.

Exercise

Reconciliation by value

Solve it in your editor.

The reconciliation above only checks whether the row is there. Write Reconcile, which compares two tables with totals in cents and returns one line for each difference: a row only in the old one, a row only in the new one, or a different total. The numbers are in order.

The old table: 1004 → 5000, 1011 → 4250, 1012 → 9900. The new one: 1011 → 4250, 1012 → 9990, 1013 → 1800.

1004: only in the old one
1012: old 9900, new 9990
1013: only in the new one
mismatches: 3

More tasks

By customer. Change the Router to choose by the number of the customer instead of the number of the order. Add a field for the customer and run two orders of one customer at 50%.
Going back. Run the plan 0, 10, 50, 10, 100 and check that at the second 10% the orders are the same as at the first.

Test

Check yourself

1. What does a strangler fig do?It puts a go-between in front of the old system and moves part of the traffic to the new one in steps, until the old one gets nothing and is switched offIt keeps the old and the new system side by side foreverIt rewrites the system at once and releases it in one night

2. Why does the new one first run in the shadow?To compare its answers with the old one on the real orders, before any customer sees one of its answersTo be fasterSo that its server costs nothing

3. The Router sends a random 10% of the requests to the new one. What is the problem?The same order goes sometimes to the old one and sometimes to the new one, and the customer sees different totals every timeRandom numbers are slowThere is no problem: 10% is 10%

4. Why does backfill not write over rows that are already in the new table?Because it is faster that wayBecause those rows came from the dual write and are newer than the copy of the historyBecause the new table is read-only

5. When is the old table removed (contract)?Already at expand, so there are not two tablesLast, when reconciliation has shown 0 mismatches for a long time, reads have long been from the new one, and no code writes to the old oneRight after the backfill

Changing a live system (strangler fig, expand–contract)

Cookies