Skip to content
wannacode

Changing a live system (strangler fig, expand–contract)

In short: Strangler fig replaces an old system step by step with a router in front that moves traffic to the new one, while expand–contract changes the data with dual writes, backfill and reconciliation.

Problem

The big switch-over

The orders module is from 2016. It computes the total: under 50 € the customer pays 5 € delivery, from 50 € up the delivery is free. The team writes a new module, with new code and a new table. The plan: on Saturday night the site is stopped, the old one is switched off, the new one is switched on.

Think about what happens if the new one has a bug that the tests did not catch. On Monday customers in three countries see wrong totals. Going back to the old one is hard by now: since Saturday the new one writes orders into the new table, and the old one does not know them. The choice is between bad and worse: fix it live, or go back and lose orders.

The problem is not the new code. The problem is that the switch-over is one big step with no way back.

The idea

Strangler fig: the new one grows around the old

The name is a plant's: the strangler fig sprouts on another tree, sends roots down around it and over the years replaces it. The tree disappears, and the shape stays.

A system is replaced the same way. A go-between (here the Router) is put in front of the old one, and the rest of the shop talks to it. At first it sends everything to the old one. Then the new one takes a small part, then a bigger one, finally everything. The old one is switched off last, when it no longer gets anything. This is a strangler fig.

Each step is small and has a way back: if the new one misbehaves, the percent is lowered. The Router is the same kind of place as an anti-corruption layer: a boundary behind which whatever is there can be replaced without the others noticing.

See it

The Router in code

Both modules answer the same question: how much the order costs.

@dataclass(frozen=True)
class Order:
    id: int
    subtotal: int
# the old module (2016)
class Legacy:
    def total(self, order):
        s = order.subtotal
        return s if s >= 50 else s + 5
FREE_FROM = 50
DELIVERY = 5
class Fresh:
    def total(self, order):
        free = order.subtotal >= FREE_FROM
        return order.subtotal + (0 if free else DELIVERY)

Python wants no interface: it is enough that Legacy and Fresh have a method total. The Router is such a module too:

class Router:
    def __init__(self, legacy, fresh, percent):
        self.legacy = legacy
        self.fresh = fresh
        self.percent = percent
        self.by_legacy = 0
        self.by_fresh = 0
        self.mismatches = []


    def total(self, order):
        # the same order always goes to the same place
        if order.id % 10 * 10 < self.percent:
            self.by_fresh += 1
            return self.fresh.total(order)
        self.by_legacy += 1
        answer = self.legacy.total(order)
        # shadow: the new one computes, never answers
        if self.fresh.total(order) != answer:
            self.mismatches.append(order.id)
        return answer

The order goes to the new one if its last digit times 10 is below the percent: at 10% only the digit 0, at 50% the digits 0 to 4. When the old one answers, the new one computes the same order in the shadow: its result is only compared, the customer does not see it.

See it

In the shadow

The first version of the new one:

# the first version of the new one: free delivery forgotten
class FreshDraft:
    def total(self, order):
        return order.subtotal + DELIVERY

Ten orders, at 0%:

orders = [
    Order(1001, 30), Order(1002, 64), Order(1003, 18), Order(1004, 45),
    Order(1005, 120), Order(1006, 9), Order(1007, 50), Order(1008, 22),
    Order(1009, 75), Order(1010, 40),
]
router = Router(Legacy(), FreshDraft(), 0)
for o in orders:
    router.total(o)
print(f"0%: old {router.by_legacy}, new {router.by_fresh}")
print(f"mismatches: {router.mismatches}")
Output
0%: old 10, new 0
mismatches: [1002, 1005, 1007, 1009]

All 10 are answered by the old one, and all totals are right. And the list says that the new one differs on 4 orders, all of them from 50 € up: it forgot the free delivery. The bug is found on the real orders, without a single customer seeing it. The fix is Fresh from above.

class Router:
    def __init__(self, legacy, fresh, percent):
        self.legacy = legacy
        self.fresh = fresh
        self.percent = percent
        self.by_legacy = 0
        self.by_fresh = 0
        self.mismatches = []


    def total(self, order):
        # the same order always goes to the same place
        if order.id % 10 * 10 < self.percent:
            self.by_fresh += 1
            return self.fresh.total(order)
        self.by_legacy += 1
        answer = self.legacy.total(order)
        # shadow: the new one computes, never answers
        if self.fresh.total(order) != answer:
            self.mismatches.append(order.id)
        return answer
Step by step, in words
  1. Router stands in front of the old orders module (2016) and the new one. The shop talks only to Router; it decides who answers.
  2. 0%: the old one answers all 10. The new one computes in the shadow and differs in 4: 1002, 1005, 1007, 1009. All are over 50 €: it forgot the free delivery.
  3. Fixed and still at 0%: 0 mismatches. Customers have not seen a single wrong total.
  4. 10%: the order whose last digit is 0 (1010) goes to the new one. The same order always goes the same way, so a customer does not jump between the two.
  5. 50%: 5 of 10 to the new one. If it misbehaves, the way back is one number: 10 again, and the same orders return to the old one.
  6. 100%: the new one answers everything. The old one gets no traffic, but it still runs and still has its data.
  7. Switching the old one off is the point of no return. It comes last, after weeks with 0 mismatches, and with its data archived.

See it

Step by step

With the fixed new one, through the whole plan:

orders = [
    Order(1001, 30), Order(1002, 64), Order(1003, 18), Order(1004, 45),
    Order(1005, 120), Order(1006, 9), Order(1007, 50), Order(1008, 22),
    Order(1009, 75), Order(1010, 40),
]
legacy = Legacy()
fresh = Fresh()
for percent in [0, 10, 50, 100]:
    router = Router(legacy, fresh, percent)
    for o in orders:
        router.total(o)
    print(f"{percent:3}%: new {router.by_fresh}, old {router.by_legacy}, mismatches {len(router.mismatches)}")
Output
  0%: new 0, old 10, mismatches 0
 10%: new 1, old 9, mismatches 0
 50%: new 5, old 5, mismatches 0
100%: new 10, old 0, mismatches 0

The mismatches stay 0 at 10% and at 50% too. In a real shop days pass between two steps: each step is watched until the errors, the response time and the complaints show that it is calm.

The choice is by the number of the order, not random. So an order always goes to the same place, and if the percent is lowered from 50 to 10, the same orders go back to the old one. In a real project the choice is by something that keeps the customer in one place: the number of the customer or of the basket.

The percent is a setting (a flag), not code: in a real project it changes without a new release, so going back takes seconds too.

Try it

Predict the output

percent = 30
for id in [1007, 1012, 1020]:
    print(id, "new" if id % 10 * 10 < percent else "old")
Answer

Try it

Find the bug

A colleague writes a shorter Router: "10% is 10%, let it be random". Customers start writing that the total in the basket changes on every reload. Which line is the bug on?

Pick the line with the bug.

The idea

The data: expand–contract

The code is replaced with the Router. The data is harder: the new one wants a new table. In the old one the total is text with a decimal comma ("64,00"), in the new one it is a number in cents (6400). And while the traffic moves, both tables have to be right.

In the lesson on migrations changing a column was done in steps: expand, transition, contract. Here it is the same, but for a whole table and with a check:

  1. Expand: the new table is added next to the old one. Nothing is removed.
  2. Dual write: every new order is written into both.
  3. Backfill: the history (the orders from before the dual write) is copied into the new one.
  4. Reconciliation: the two are compared until the mismatches are 0.
  5. Reads move to the new one, the old one is no longer written.
  6. Contract: the old table is removed.

This is expand–contract. Up to step 6 each step can be undone: the old table is still there and it is right.

See it

Dual write, backfill and reconciliation

The two tables here are dictionaries, not a database, so that the logic can be seen:

def cents(text):
    euros, rest = text.split(",")
    return int(euros) * 100 + int(rest)




class Migration:
    def __init__(self):
        # the old table (2016): the total is text with a decimal comma
        self.old_rows = {}
        # the new table: the total is in cents
        self.new_rows = {}


    # dual write: every new order goes into both
    def place(self, id, total):
        self.old_rows[id] = total
        self.new_rows[id] = cents(total)


    # backfill: copies the history without writing over new rows
    def backfill(self):
        for id, total in self.old_rows.items():
            try:
                self.new_rows.setdefault(id, cents(total))
            except ValueError:
                print(f'✗ backfill {id}: "{total}"')


    # reconciliation: every old row must be in the new table too
    def reconcile(self):
        return [id for id in self.old_rows if id not in self.new_rows]
m = Migration()
# the history: orders from before the migration, only in the old table
m.old_rows[1001] = "35,00"
m.old_rows[1002] = "64,00"
m.old_rows[1003] = "23,00"
m.old_rows[1004] = "5O,00"
# from today on: dual write
m.place(1011, "42,50")
m.place(1012, "99,90")
m.backfill()
missing = m.reconcile()
print(f"old: {len(m.old_rows)}, new: {len(m.new_rows)}, missing in the new one: {missing}")
print("reads can move to the new table" if not missing else "not yet: fix the difference first")
Output
✗ backfill 1004: "5O,00"
old: 6, new: 5, missing in the new one: [1004]
not yet: fix the difference first

Order 1004 from 2016 has the letter O instead of a zero in its total. The old module lives with that, but turning it into cents cannot. Backfill does not hide it: it says it out loud and goes on. Reconciliation catches it and says "not yet". Someone will fix the row in the old table, backfill will run again, and reconciliation will show 0.

Backfill can be run as many times as needed: it copies only what is missing. The dual write can be done as two writes in one transaction (the lesson on databases) or through an outbox, so that no write ends up in only one of the tables.

Try it

Fill in the blank

Backfill must not write over a row that is already in the new table. How?

for id, total in self.old_rows.items():
    self.new_rows.(id, cents(total))

Try it

Put the steps in order

  1. stop_old_writes() # the old one read-only
  2. reconcile() # compare until the mismatches are 0
  3. switch_reads() # reads from the new one
  4. expand() # the new table next to the old one
  5. backfill() # the history into the new one
  6. contract() # drop the old one
  7. dual_write() # every write into both

The idea

The point of no return

Every step so far has a way back: the percent is lowered, reads go back to the old table, the dual write goes on. Two steps do not: switching off the old module and removing the old table. After them, going back means restoring from an archive.

So they come last and are decided on purpose. Before them: weeks with 0 mismatches in reconciliation, no code that reads or writes the old one, and an archive of the data. It is good to write down the plan, the criterion for each step and how it is undone in an ADR before starting: halfway through a migration everyone wants to finish it faster.

A migration stopped halfway is the worst: two systems, two sets of data and a dual write, all kept alive forever. The plan has to reach contract.

The idea

When it hurts

  • A small system with little data. If stopping for half an hour at night is acceptable and the data moves in minutes, a Router, a shadow and reconciliation are weeks of work with no gain. One migration with a plan for going back (an archive before it) is enough.
  • When there is nowhere to put a Router. If the old one is called directly from twenty places, first there has to be one boundary in front of it (a facade, a hub), and only then can it be strangled.
  • The shadow is not free. The new one computes everything once more. If it sends emails or takes money, that must not happen in the shadow: the shadow has to be free of side effects.
  • An endless "almost". If 100% never comes (the last 5% are the hardest), two systems remain to maintain. Without a deadline and a criterion, a strangler fig turns into two monoliths instead of one.
More precisely, if you are curious

The name comes from an article by Martin Fowler from 2004 (Strangler Fig Application). Expand–contract is also described as parallel change.

The Router in a real project is most often in a gateway or load balancer in front of the two systems: a rule by path (/orders to the new one) or by percent. They are deployed and monitored as infrastructure, and that is in the DevOps course ZeroToOps. The percent is often a feature flag: a setting that changes without a new release (LaunchDarkly, Unleash, OpenFeature).

Schema migrations are written with tools like Flyway and Liquibase (Java), EF Core Migrations (.NET), Alembic (Python), Prisma or Knex (JavaScript). They run the steps in order, but they do not decide when: that is the expand–contract plan.

Instead of a dual write in the code CDC (change data capture) is sometimes used: a tool like Debezium reads the log of the old database and carries each change into the new one. The code is not touched, but one more system is added to watch.

The shadow comparison at large scale is done asynchronously: the answers are recorded and compared later, so that the new one does not slow the customer down.

Exercise

Reconciliation by value

Solve it in your editor.

The reconciliation above only checks whether the row is there. Write reconcile, which compares two tables with totals in cents and returns one line for each difference: a row only in the old one, a row only in the new one, or a different total. The numbers are in order.

The old table: 1004 → 5000, 1011 → 4250, 1012 → 9900. The new one: 1011 → 4250, 1012 → 9990, 1013 → 1800.

Output
1004: only in the old one
1012: old 9900, new 9990
1013: only in the new one
mismatches: 3

More tasks

  • By customer. Change the Router to choose by the number of the customer instead of the number of the order. Add a field for the customer and run two orders of one customer at 50%.
  • Going back. Run the plan 0, 10, 50, 10, 100 and check that at the second 10% the orders are the same as at the first.

Test

Check yourself

1. What does a strangler fig do?
2. Why does the new one first run in the shadow?
3. The Router sends a random 10% of the requests to the new one. What is the problem?
4. Why does backfill not write over rows that are already in the new table?
5. When is the old table removed (contract)?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·