Tests as a safety net

Code in:JavaC#PythonJavaScript

In short: Characterization tests pin down what old code does today, so they catch any accidental change in behaviour while you clean it up.

Problem

The harmless change

The shop works out delivery with the function delivery_fee. It was written three years ago and has no tests. Delivery is €4.50, outside Sofia it is €2 more, to Vidin another €3, and above 10 kg you pay €0.50 for every extra kilogram. From €50 up it is free.

def delivery_fee(total, kg, city):
    fee = 4.50
    if city != "Sofia":
        fee = fee + 2.00
    if city == "Vidin":
        fee = fee + 3.00
    if kg > 10:
        fee = fee + (kg - 10) * 0.50
    if total >= 50:
        fee = 0
    return fee

Last month a colleague "tidied it up": if the order is free, why calculate anything, so the check moved to the top. They wrote total > 50 instead of total >= 50. For a whole week, orders of exactly €50 paid for delivery. Nobody noticed until the complaints came in.

Now you need to add a new city. Do you touch this function? If you don't, every new rule gets stuck on top and the code gets scarier and scarier. If you do, you might break something that works and never find out.

The idea

A net under the rope

A tightrope walker walks calmly because there is a net below. If they fall, the net catches them before anything bad happens.

For code, the net is tests. Before you change how the code is written, you write tests that say what it does today. Then you change it and run the tests. Green: the behaviour is the same. Red: something has changed, and you know exactly what.

Tests like these are called characterization tests: they don't check whether the code is right, they describe how it behaves. If the code does something odd today, the test records the odd part too. This is the safety net: no code gets touched until there are tests under it.

See it

Write down what it does today

You don't guess what the function ought to return. You ask it. A short program calls it with a few orders and prints the answers:

print("€20, 2 kg, Sofia → ", delivery_fee(20, 2, "Sofia"), sep="")
print("€20, 2 kg, Plovdiv → ", delivery_fee(20, 2, "Plovdiv"), sep="")
print("€20, 16 kg, Sofia → ", delivery_fee(20, 16, "Sofia"), sep="")
print("€50, 2 kg, Plovdiv → ", delivery_fee(50, 2, "Plovdiv"), sep="")
print("€60, 25 kg, Plovdiv → ", delivery_fee(60, 25, "Plovdiv"), sep="")

Output

€20, 2 kg, Sofia → 4.5
€20, 2 kg, Plovdiv → 6.5
€20, 16 kg, Sofia → 7.5
€50, 2 kg, Plovdiv → 0
€60, 25 kg, Plovdiv → 0

The last line is a surprise. The parcel is 25 kg, and delivery is free: free delivery also drops the weight surcharge. Is that a bug? Maybe. But that is how it works today, customers rely on it, and whether to change it is not your call while you are tidying the code. You write it down as it is.

Each line of the probe becomes a test:

import unittest

from delivery import delivery_fee


class DeliveryTest(unittest.TestCase):
    def test_sofia(self):
        self.assertEqual(delivery_fee(20, 2, "Sofia"), 4.5)

    def test_outside_sofia(self):
        self.assertEqual(delivery_fee(20, 2, "Plovdiv"), 6.5)

    def test_heavy(self):
        self.assertEqual(delivery_fee(20, 16, "Sofia"), 7.5)

    def test_free_from_50(self):
        self.assertEqual(delivery_fee(50, 2, "Plovdiv"), 0)

    def test_free_even_when_heavy(self):
        # That is how it works today: free delivery also drops the weight surcharge.
        self.assertEqual(delivery_fee(60, 25, "Plovdiv"), 0)

Result

PASS  test_free_even_when_heavy
PASS  test_free_from_50
PASS  test_heavy
PASS  test_outside_sofia
PASS  test_sofia
Tests: 5, passed: 5, failed: 0

Five tests, five green. The net is up.

More precisely, if you are curious

The tests here run the same way as in the testing lesson of the module "Programs in the real world". The checked program puts the function and the test in one file (without the from delivery import ... line) and runs them with unittest, sorting the results by name. That way the output is the same on every run.

The values in the test are what Python printed. The last 0 is an integer, because the old code writes fee = 0. For assertEqual, 0 == 0.0 is true, but the probe shows exactly what the function returns.

See it

The harmless change under the net

Now the colleague makes the same change, but under a net. The check goes to the top:

def delivery_fee(total, kg, city):
    if total > 50:
        return 0
    fee = 4.50
    if city != "Sofia":
        fee = fee + 2.00
    if city == "Vidin":
        fee = fee + 3.00
    if kg > 10:
        fee = fee + (kg - 10) * 0.50
    return fee

Result

FAIL  test_free_from_50: 6.5 != 0
PASS  test_free_even_when_heavy
PASS  test_heavy
PASS  test_outside_sofia
PASS  test_sofia
Tests: 5, passed: 4, failed: 1

One test is red, and it says exactly which case: an order of €50 outside Sofia. Expected 0, got €6.50. The bug is caught at the desk, not by a customer.

class DeliveryTest(unittest.TestCase):
    def test_sofia(self):
        self.assertEqual(delivery_fee(20, 2, "Sofia"), 4.5)

    def test_outside_sofia(self):
        self.assertEqual(delivery_fee(20, 2, "Plovdiv"), 6.5)

    def test_heavy(self):
        self.assertEqual(delivery_fee(20, 16, "Sofia"), 7.5)

    def test_free_from_50(self):
        self.assertEqual(delivery_fee(50, 2, "Plovdiv"), 0)

    def test_free_even_when_heavy(self):
        # That is how it works today: free delivery also drops the weight surcharge.
        self.assertEqual(delivery_fee(60, 25, "Plovdiv"), 0)

Step by step, in words

1. Five tests, one for each line of the probe. Each one calls delivery_fee and compares with the answer it gives today.
2. delivery_fee(20, 2, "Sofia") returns 4.5. The test pins it down.
3. delivery_fee(20, 2, "Plovdiv") returns 6.5. The test pins it down.
4. delivery_fee(20, 16, "Sofia") returns 7.5. The test pins it down.
5. delivery_fee(50, 2, "Plovdiv") returns 0. The test pins it down.
6. delivery_fee(60, 25, "Plovdiv") returns 0, although the parcel is 25 kg. Odd, but that is how it is today: the test keeps it, it doesn't fix it.
7. A colleague moves the free delivery check to the top and writes > instead of >=. The whole net runs again.
8. test_free_from_50 fails: expected 0, but got 6.5. An order of exactly €50 now pays for delivery.
9. With >= all five tests are green. The change is safe: the net has checked it.

With >= everything is green:

def delivery_fee(total, kg, city):
    if total >= 50:
        return 0
    fee = 4.50
    if city != "Sofia":
        fee = fee + 2.00
    if city == "Vidin":
        fee = fee + 3.00
    if kg > 10:
        fee = fee + (kg - 10) * 0.50
    return fee

Result

PASS  test_free_even_when_heavy
PASS  test_free_from_50
PASS  test_heavy
PASS  test_outside_sofia
PASS  test_sofia
Tests: 5, passed: 5, failed: 0

The change is the same one that went unnoticed for a whole week. The only difference is the net.

Try it

Predict the output

Someone changes the line for the weight surcharge and forgets to subtract the first 10 kg:

fee = fee + kg * 0.50

The tests from the card "Write down what it does today" run again. What happens?

def delivery_fee(total, kg, city):
    fee = 4.50
    if city != "Sofia":
        fee = fee + 2.00
    if city == "Vidin":
        fee = fee + 3.00
    if kg > 10:
        fee = fee + kg * 0.50
    if total >= 50:
        fee = 0
    return fee

Answer

Only test_heavy fails

All five pass: the change is small

test_heavy and test_free_even_when_heavy fail

Check

Try it

What you write as expected

For €60, 25 kg and Plovdiv the probe printed: 0. You are sure the right delivery is €14. What do you write in the test before the change?

def test_free_even_when_heavy(self):
    self.assertEqual(delivery_fee(60, 25, "Plovdiv"), )

Check

The idea

Small steps

The net helps most when you fall from low down. If you change twenty things at once and one test fails, you don't know which of the twenty is to blame. If you change one thing and the test fails, the culprit is obvious.

So the work has a rhythm: one small change, run the tests, green, the next one. Red means: undo the last change (Ctrl+Z, or git checkout on the file) and make it in even smaller pieces. That way the code is in a working state almost all the time.

Put the steps in order:

1. Run the tests again↑↓
2. Green: keep it (commit). Red: undo only the last change↑↓
3. Add tests for the path you are about to touch↑↓
4. Make one small change↑↓
5. Run the tests: all of them are green↑↓

Check

Try it

Find the bug

Someone wrote the test for €50 like this. Then the colleague makes the change with total > 50 from the card "The harmless change under the net":

class DeliveryTest(unittest.TestCase):
    def test_sofia(self):
        self.assertEqual(delivery_fee(20, 2, "Sofia"), 4.5)

    def test_outside_sofia(self):
        self.assertEqual(delivery_fee(20, 2, "Plovdiv"), 6.5)

    def test_heavy(self):
        self.assertEqual(delivery_fee(20, 16, "Sofia"), 7.5)

    def test_free_from_50(self):
        print(delivery_fee(50, 2, "Plovdiv"))

    def test_free_even_when_heavy(self):
        # That is how it works today: free delivery also drops the weight surcharge.
        self.assertEqual(delivery_fee(60, 25, "Plovdiv"), 0)

Result

6.5
PASS  test_free_even_when_heavy
PASS  test_free_from_50
PASS  test_heavy
PASS  test_outside_sofia
PASS  test_sofia
Tests: 5, passed: 5, failed: 0

The wrong amount is printed, and the tests are green. Which line is the bug on?

Pick the line with the bug.

1class DeliveryTest(unittest.TestCase):2 def test_sofia(self):3 self.assertEqual(delivery_fee(20, 2, "Sofia"), 4.5)45 def test_outside_sofia(self):6 self.assertEqual(delivery_fee(20, 2, "Plovdiv"), 6.5)78 def test_heavy(self):9 self.assertEqual(delivery_fee(20, 16, "Sofia"), 7.5)1011 def test_free_from_50(self):12 print(delivery_fee(50, 2, "Plovdiv"))1314 def test_free_even_when_heavy(self):15 # That is how it works today: free delivery also drops the weight surcharge.16 self.assertEqual(delivery_fee(60, 25, "Plovdiv"), 0)

Check

The idea

What the net doesn't catch

The net catches only where it is stretched. There is no test for Vidin. Someone "tidies up" the cities with else if:

def delivery_fee(total, kg, city):
    fee = 4.50
    if city == "Vidin":
        fee = fee + 3.00
    elif city != "Sofia":
        fee = fee + 2.00
    if kg > 10:
        fee = fee + (kg - 10) * 0.50
    if total >= 50:
        fee = 0
    return fee

The tests

PASS  test_free_even_when_heavy
PASS  test_free_from_50
PASS  test_heavy
PASS  test_outside_sofia
PASS  test_sofia
Tests: 5, passed: 5, failed: 0

Vidin before and after

Vidin before: 9.5
Vidin after: 7.5

The tests are green, and delivery to Vidin got €2 cheaper: before, Vidin paid both the €2 for outside Sofia and €3 on top. No test goes down this path, so no test fails.

Before you touch code, check which paths have tests and add some for the ones you will touch. That is what coverage shows: which lines ran while the tests were running.

The coverage.py package: coverage run -m unittest, then coverage report -m shows the lines without a test. In PyCharm it is "Run with Coverage".

In Python there is one more thing: there is no compiler that reads the whole file. Someone renames fee to price by hand and forgets it in the Vidin branch:

def delivery_fee(total, kg, city):
    price = 4.50
    if city != "Sofia":
        price = price + 2.00
    if city == "Vidin":
        price = fee + 3.00
    if kg > 10:
        price = price + (kg - 10) * 0.50
    if total >= 50:
        price = 0
    return price

The five tests are still green: none of them reaches this line. The error shows up only when someone orders to Vidin:

try:
    delivery_fee(20, 2, "Vidin")
except NameError as error:
    print("Vidin:", error)

Output

Vidin: name 'fee' is not defined

That is why tests protect even more here than in Java and C#.

More precisely, if you are curious

100% coverage doesn't mean every case is checked. The line with total >= 50 is "covered" by a test with €60 too, but exactly €50 is a separate case, and only the €50 test catches >. Coverage tells you where the net has holes. How dense it is where it exists depends on the cases you picked: the boundaries (€50, 10 kg), every branch of an if, and the odd values from the probe.

The coverage commands are not run here: the site's checks run without a network and without external packages.

Exercise

A net for Vidin

Solve the task in your editor.

A net for Vidin. Add two tests to the tests from the card "Write down what it does today": an order of €20, 2 kg to Vidin and an order of €50, 2 kg to Vidin. Run the probe first to see what the code returns today, and pin that down.

Result

PASS  test_free_even_when_heavy
PASS  test_free_from_50
PASS  test_heavy
PASS  test_outside_sofia
PASS  test_sofia
PASS  test_vidin
PASS  test_vidin_free_from_50
Tests: 7, passed: 7, failed: 0

Then make the change with else if from the card "What the net doesn't catch". Now it has to fail:

Result

FAIL  test_vidin: 7.5 != 9.5
PASS  test_free_even_when_heavy
PASS  test_free_from_50
PASS  test_heavy
PASS  test_outside_sofia
PASS  test_sofia
PASS  test_vidin_free_from_50
Tests: 7, passed: 6, failed: 1

More tasks

• The weight boundary. Add a test for exactly 10 kg and for 11 kg. Then change kg > 10 to kg >= 10. Does anything fail? Why?
• The new city. Add Varna with €1 on top. First a test for Varna that fails, then the rule, then all the tests again.
• A probe for everything. Write the probe as a loop over a table of cases (amount, weight, city), so that a new case is a new row in the table.

Test

Check yourself

1. The probe shows that an order of €60 with a 25 kg parcel is delivered for free. You think this is a bug. What do you write in the test before the change?

The value that ought to be rightThe value the function returns today, with a note that it looks oddNothing: you don't write a test for a case like that

2. Five tests pass. You rearrange the function and one fails. What do you do?

Change the expected value in the test so that it passesDelete the test, since it is in the wayUndo the last change and make it in smaller steps

3. Why are the steps small, with the tests run after each one?

To have more commitsWhen a test fails, you know the last small change is to blameBecause tests are fast only for small code

4. All tests are green. Does that mean the change didn't change anything?

Yes, if there are more than tenNo: tests protect only the cases they check. A path without a test may have changedYes, since they are green

5. A test calls the function and prints the result without comparing it. What does it protect?

Only the case it printsNothing: it passes with any result, unless the program stops with an errorEverything it prints

Check

Start the lesson over

Related lessons

• Builds on: Tests with unittest · Names in code · Functions: one job, few parameters, no hidden side effects · Code smells: long method, large class, duplication, primitives instead of types, long if chains
• Continues in: Refactoring in small steps · Exercise: a messy class, cleaned up step by step · Dependencies: other people’s code that yours runs
• Tasks on this topic: Clean code and refactoring