Software architecture project

Problem

The task

The shop owner writes:

From the spring we also sell in Greece and Croatia. We are also letting in outside sellers: they upload their own products, we take a commission. Black Friday must not go down like last year. How much will it cost and when?

This is the architect's real first task. Nobody tells you "build a modular monolith". They tell you a wish, and you have to bring back:

requirements with numbers that can be measured;
at least two options with their cost;
a decision, recorded in an ADR;
a migration plan for the live system, reversible at every step;
a risk register;
a check that guards the new boundaries.

The three countries are in the eurozone, so the money stays in euros. Everything else differs: language, VAT, couriers, laws on returns.

The idea

The plan

Each step uses a lesson from the module:

The requirements with percentiles and an availability budget, as in "Non-functional requirements".
The options with the trade-offs from "Trade-offs" and the shapes from "Monolith, modular monolith, services".
An ADR as in "ADR: architecture decision records".
The migration with strangler fig and expand–contract.
The risks as in "Architecture review".
A fitness function for the boundaries, also from the review.

Each step has an output to compare yours with. The solutions are under "Solution" after each step, and at the end there is a sample solution of the whole task. Try it yourself first: architecture is learned by deciding, not by reading decisions.

Exercise

Step 1: requirements with numbers

"Make it fast and don't let it go down" is not a requirement. Turn each of these wishes into a quality scenario: source, stimulus, response and a measure with a number.

"Black Friday must not go down."
"Payment must always work."
"Sellers should see their sales right away."
"A new country should be easy to launch."

Sample scenarios

Quality	Source and stimulus	Response	Measure
scale	Black Friday, 5 times the usual traffic	the payment page responds	p95 under 800 ms, p99 under 2 s
availability	a buyer in any of the three countries pays	the payment goes through	99.7% of the minutes in a month, at most 130 min of downtime
consistency	a seller changes stock	buyers see it	within 2 min; anything sold over stock is cancelled with an apology within 1 hour
isolation	a seller uploads 10,000 products at once	the shop works as before	catalogue p95 grows by no more than 20%
extensibility	a fourth country	it launches with settings	within 2 weeks, with no change to the orders code

The numbers here are examples: in a real task they come from the owner ("what does an hour of downtime on Black Friday cost?") and from measuring today's system.

"Sellers should see their sales right away" became "within 2 minutes". That is a deliberate choice of eventual consistency: "right away" would put the sellers' systems on the payment path.

Exercise

Step 1b: the budget of the chain

Payment goes through the shop API, the database and the payment provider. The first idea for sellers is to ask the seller's system, on every payment, whether the item is in stock. The availabilities and p95s below are examples (for providers from the contract, for sellers from a trial):

chain = [
    Hop("shop API", 99.95, 120),
    Hop("database", 99.95, 40),
    Hop("payment", 99.9, 450),
    Hop("stock at the sellers", 99.0, 400),
]
budget("Stock at the sellers is on the payment path:", chain, 99.7, 800)
budget("Without it (a copy, updated from events):", chain[:3], 99.7, 800)

Write budget, which multiplies availability along the chain, adds up the p95s and compares with the budget: 99.7% (downtime in minutes for a 30-day month) and p95 under 800 ms. Run it with the sellers on the chain and without them:

Stock at the sellers is on the payment path:
  availability 98.80% → 517 min of downtime a month (budget 130) ✗
  p95 1010 ms (budget 800) ✗
Without it (a copy, updated from events):
  availability 99.80% → 86 min of downtime a month (budget 130) ✓
  p95 610 ms (budget 800) ✓

Solution

@dataclass
class Hop:
    name: str
    availability: float  # %
    p95: int  # ms

MINUTES_PER_MONTH = 30 * 24 * 60


def budget(title, chain, target, p95_budget):
    availability = 1.0
    p95 = 0
    for hop in chain:
        availability *= hop.availability / 100
        p95 += hop.p95  # the sum of p95s is an upper bound, not the exact p95 of the chain
    down = round((1 - availability) * MINUTES_PER_MONTH)
    allowed = round((1 - target / 100) * MINUTES_PER_MONTH)
    print(title)
    mark = "✓" if down <= allowed else "✗"
    print(f"  availability {availability * 100:.2f}% → {down} min of downtime a month (budget {allowed}) {mark}")
    print(f"  p95 {p95} ms (budget {p95_budget}) {'✓' if p95 <= p95_budget else '✗'}")

One system at 99.0% drags payment down to 98.80%: 517 minutes of downtime a month against a budget of 130. Latency adds up as well. So stock at the sellers cannot be a synchronous step of payment. It becomes a copy on our side, updated from events (the lesson "Events and queues"), and the cost is that the copy can lag. This decision comes from the numbers, not from taste.

The sum of the p95s is an upper bound: the p95 of the chain is usually lower, because all the steps are rarely slow at once. For a budget, the upper bound is the safe side.

Exercise

Step 2: options with their cost

Describe at least two options that would really work, and the cost of each: money, time, people and what you lose. Three natural options:

	A: three copies of the shop	B: a modular monolith	C: services from day one
idea	one shop per country	the country is a setting, sellers are a module	catalogue, orders, sellers, payment separately
time to launch	fastest	medium	slowest
cost per month	three times the infrastructure and upkeep	one system	many services, monitoring, network
what you lose	the copies drift apart and never merge back	a change for one country ships to all three	the team learns distributed systems while shipping

The comparison is made in a decision matrix: criteria with weights (how important each one is) and a score from 1 to 5 for each option. The weights and scores belong to the team and the owner:

# time to launch 3, availability 3, cost per month 2, the team knows it 2, reversibility 2
weights = [3, 3, 2, 2, 2]
options = {
    "A: three copies of the shop": [5, 3, 2, 4, 2],
    "B: a modular monolith, the country is a setting": [3, 4, 4, 4, 4],
    "C: services from day one": [1, 4, 2, 2, 2],
}
for name, scores in options.items():
    print(f"{name}: {total(scores, weights)}")
print("Choice:", best(options, weights))

Write total (the sum of weight × score) and best:

A: three copies of the shop: 40
B: a modular monolith, the country is a setting: 45
C: services from day one: 27
Choice: B: a modular monolith, the country is a setting

Solution

def total(scores, weights):
    return sum(w * s for w, s in zip(weights, scores))


def best(options, weights):
    return max(options, key=lambda name: total(options[name], weights))

The matrix does not make the decision for you. It makes the weights visible: if the owner says "time is everything", that is a number you can argue about.

Exercise

Step 2b: how fragile is the decision

The scores are judgement. Check the sensitivity: raise the weight of time to launch from 3 upwards and find the first weight at which another option wins:

With weight 6 for time, the winner is A: three copies of the shop (55 against 54)

The choice holds until time becomes twice as important as availability. That is what you tell the owner: "if launching by spring matters more than anything else, the copies win, but here is their cost".

Exercise

Step 3: the ADR

Record the decision in an ADR. The check from the ADR lesson looks at whether the sections are there. Here you make it stricter, because an ADR with no cost is an advert:

at least two options under "Options";
at least one consequence that starts with "- bad:";
"Revisit when" contains a number that can be measured.

Here is the draft a colleague sends you:

# ADR-014: Three countries and sellers in the modular monolith
## Status
proposed
## Context
The shop launches in Greece and Croatia and lets outside sellers in. Payment: 99.7% a month, p95 under 800 ms.
## Options
- B: a modular monolith, the country is a setting, sellers are a module
## Decision
B. Stock at the sellers arrives asynchronously and is not on the payment path.
## Consequences
- good: one code base for the three countries

Write check_adr and run it on the draft and on your final ADR:

Draft: 3
  ✗ missing section 'Revisit when'
  ✗ Options: 1, at least 2 are needed
  ✗ Consequences: not a single 'bad:'
Final: ✓ OK

Solution

REQUIRED = ["Status", "Context", "Options", "Decision", "Consequences", "Revisit when"]


def sections(adr):
    result, current = {}, None
    for line in adr.strip().split("\n"):
        if line.startswith("## "):
            current = result[line[3:]] = []
        elif current is not None and line.strip():
            current.append(line)
    return result


def check_adr(adr):
    s = sections(adr)
    problems = [f"✗ missing section '{name}'" for name in REQUIRED if name not in s]
    count = sum(1 for line in s.get("Options", []) if line.startswith("- "))
    if count < 2:
        problems.append(f"✗ Options: {count}, at least 2 are needed")
    if not any(line.startswith("- bad:") for line in s.get("Consequences", [])):
        problems.append("✗ Consequences: not a single 'bad:'")
    if "Revisit when" in s and not any(ch.isdigit() for ch in " ".join(s["Revisit when"])):
        problems.append("✗ Revisit when: no number to measure")
    return problems

for name, text in [("Draft", DRAFT), ("Final", FINAL)]:
    problems = check_adr(text)
    print(f"{name}: {len(problems) if problems else '✓ OK'}")
    for p in problems:
        print("  " + p)

The final ADR is in the sample solution at the bottom.

Exercise

Step 4: the migration plan

The shop is running right now. Orders have to learn about the country without a stop and without a big switch-over. Put the steps in an order where each one can be undone until the last:

add the country column (empty, the old code ignores it)
the new code writes country, the old code still works
backfill: old orders get BG, then compare counts
switch off the old path and drop the old code
10%, 50%, 100% only on green metrics, back on red
1% of Greek traffic goes the new way (a flag)

The shares (1%, 10%, 50%, 100%) are not raised by the calendar but by metrics. Write rollout: for each share it reads the next recorded metrics. On green (at most 5 errors per 1000 and p95 under 800 ms) it moves forward, on red it moves traffic back to the previous share and tries again after the fix. The metrics are a recorded scenario, so the output is always the same:

observed = [
    Metrics(2, 610),
    Metrics(9, 700),
    Metrics(3, 640),
    Metrics(4, 690),
    Metrics(3, 700),
]
rollout([1, 10, 50, 100], iter(observed))

1%: errors 2/1000, p95 610 ms ✓
10%: errors 9/1000, p95 700 ms ✗ back to 1%
10%: errors 3/1000, p95 640 ms ✓
50%: errors 4/1000, p95 690 ms ✓
100%: errors 3/1000, p95 700 ms ✓
The old path is switched off: the point of no return.

Solution

@dataclass
class Metrics:
    errors_per_1000: int
    p95: int  # ms

    def healthy(self):
        return self.errors_per_1000 <= 5 and self.p95 <= 800

def rollout(shares, observed):
    step = 0
    while step < len(shares):
        m = next(observed)
        line = f"{shares[step]}%: errors {m.errors_per_1000}/1000, p95 {m.p95} ms"
        if m.healthy():
            print(line + " ✓")
            step += 1
        else:
            back = 0 if step == 0 else shares[step - 1]
            print(f"{line} ✗ back to {back}%")
    print("The old path is switched off: the point of no return.")

There is only one point of no return, and it is last: removing the old path. Everything before it is undone with a flag. The column is added empty (expand), and removed only when nobody reads it (contract).

Try it

Find the bug

A colleague writes the metrics check. The rollout goes smoothly, but 10% starts with 9 errors per 1000:

1%: errors 2/1000, p95 610 ms ✓
10%: errors 9/1000, p95 700 ms ✓
50%: errors 3/1000, p95 640 ms ✓
100%: errors 4/1000, p95 690 ms ✓
The old path is switched off: the point of no return.

Pick the line with the bug.

Exercise

Step 5: the risk register

Here are the risks the team collected. Write register, which sorts them (by score, on a tie by impact) and shows the measure. A high risk (12 or more) with no measure is an open hole and is shown with "✗", and a smaller one with no measure is consciously accepted.

risks = [
    Risk("VAT and invoices get mixed up between countries", 3, 5, "a test invoice for each country before 1%"),
    Risk("the courier in Croatia has no tracking API", 2, 4, ""),
    Risk("Black Friday traffic in Greece is 5 times higher", 3, 4, ""),
    Risk("the stock copy lags and we sell what is sold out", 4, 3, "a reservation at payment and a cancel with an apology"),
    Risk("the team does not know the sellers module", 3, 2, "pair programming for the first two months"),
]
register(risks)

15 VAT and invoices get mixed up between countries → a test invoice for each country before 1%
12 Black Friday traffic in Greece is 5 times higher → ✗ no measure
12 the stock copy lags and we sell what is sold out → a reservation at payment and a cancel with an apology
 8 the courier in Croatia has no tracking API → (accepted)
 6 the team does not know the sellers module → pair programming for the first two months

Solution

@dataclass
class Risk:
    name: str
    probability: int
    impact: int
    measure: str = ""

    @property
    def score(self):
        return self.probability * self.impact

def register(risks):
    for r in sorted(risks, key=lambda r: (r.score, r.impact), reverse=True):
        measure = r.measure or ("✗ no measure" if r.score >= 12 else "(accepted)")
        print(f"{r.score:2} {r.name} → {measure}")

For Black Friday the measure is a load test at 5 times the traffic before the launch in Greece, and a plan for what gets switched off first (recommendations, search by seller), so that payment survives. Scaling is from "Scaling", and switching off is like the circuit breaker from "When the neighbour is slow".

Exercise

Step 6: a fitness function for the new boundaries

The sellers module is new. If in its first month it starts reading the internals of orders, a year later it will be impossible to split out. The rule: every module has an api (the public part) and an internal; another module is called only through its api. The imports of the modules:

sellers.internal -> orders.api
sellers.internal -> orders.internal
orders.internal -> payments.api
orders.internal -> sellers.internal
catalog.internal -> sellers.api
orders.internal -> orders.db

✗ sellers.internal -> orders.internal: calls the internals of orders
✗ orders.internal -> sellers.internal: calls the internals of sellers
Violations: 2

if from_module != to_module and to_part != "":

Solution

# another module is called only through its api
def boundary_violations(imports):
    found = []
    for line in imports:
        source, target = line.split(" -> ")
        from_module, _ = source.split(".")
        to_module, to_part = target.split(".")
        if from_module != to_module and to_part != "api":
            found.append(f"{line}: calls the internals of {to_module}")
    return found

imports = [
    "sellers.internal -> orders.api",
    "sellers.internal -> orders.internal",
    "orders.internal -> payments.api",
    "orders.internal -> sellers.internal",
    "catalog.internal -> sellers.api",
    "orders.internal -> orders.db",
]
found = boundary_violations(imports)
for v in found:
    print("✗ " + v)
print("Violations:", len(found))

Run it on every commit with a baseline, as in the review. Today's two violations are fixed before the sellers module ships.

See it

Sample solution

Compare your solution with this one. It is not the only right one: any solution whose numbers, options and risks are honest and recorded is right.

The requirements. The five scenarios from step 1. The key ones: payment at 99.7% and p95 under 800 ms, and stock at the sellers eventually consistent, within 2 minutes.

The options. A, B and C with their cost; the choice is B with 45 points against 40 and 27. The decision holds up to a weight of 6 for time.

The architecture of B.

The country is a setting: language, VAT, courier and return rules come from a per-country table, not from an if in the code.
Sellers are a module in the monolith, with their own data and a public api ("Monolith, modular monolith, services").
Stock at the sellers arrives as events into a copy on our side (an outbox on their side or a queue on ours).
Anything sold over stock is cancelled with a saga: reservation, payment, and on a refusal from the seller, a refund.
Warehouse 2009 and accounting stay behind an anti-corruption layer, as before.

The ADR.

# ADR-014: Three countries and sellers in the modular monolith
## Status
accepted
## Context
The shop launches in Greece and Croatia and lets outside sellers in. Payment: 99.7% a month, p95 under 800 ms.
## Options
- A: three copies of the shop, one per country
- B: a modular monolith, the country is a setting, sellers are a module
- C: services from day one
## Decision
B. Stock at the sellers arrives asynchronously and is not on the payment path.
## Consequences
- good: one code base for the three countries
- bad: a change for one country ships to all three
- bad: the stock copy can be minutes late
## Revisit when
If the sellers grow past 500 or the teams grow past 3.

The plan. The country column (expand), the new code writes to it, a backfill of old orders as Bulgaria, then 1%, 10%, 50% and 100% of Greek traffic on green metrics, and finally removing the old path (contract). Croatia follows Greece with the same plan.

The risks. The register from step 5, with a measure for Black Friday. The owner of each risk answers for it, and the register is reviewed every two weeks until the launch.

The checks. The boundaries from step 6 and the budget from step 1 as a fitness function: a load test before every launch that fails if the p95 of payment is over 800 ms. How they run in the pipeline is a topic for ZeroToOps, the DevOps course.

What we did not decide. A separate service for sellers. The ADR says when to come back to it: more than 500 sellers or more than 3 teams.

Test

Check yourself

1. Why does stock at the sellers leave the payment path?Because it is cheaper that wayBecause sellers are slow to programThe chain multiplies: with it, payment is at 98.80% (517 min of downtime a month), and the budget is 130 min.

2. The matrix picks B. What does the sensitivity analysis check?How much the option costsHow much the weights must change for the choice to change; if a small change flips it, the decision is fragileWhether the scores are right

3. What makes an ADR useful two years later?At least two real options, honest "bad" consequences and a condition with a number for when the decision is revisitedOnly the decision, without the options, to keep it shortThe longer, the better

4. Rolling out 10% shows 9 errors per 1000 with a limit of 5. What does the plan do?Moves everything back to the old system for goodGoes on to 50%, because p95 is goodMoves traffic back to the previous share (1%), fixes it and tries 10% again

5. When does the point of no return come in the plan?When the country column is addedAt the first 1%Only after 100% on green metrics, when the old path and its code are removed

Your first architecture task

Cookies