Tests as a safety net

Code in:JavaC#PythonJavaScript

In short: Characterization tests pin down what old code does today, so they catch any accidental change in behaviour while you clean it up.

Problem

The harmless change

The shop works out delivery with the method Delivery.fee. It was written three years ago and has no tests. Delivery is €4.50, outside Sofia it is €2 more, to Vidin another €3, and above 10 kg you pay €0.50 for every extra kilogram. From €50 up it is free.

class Delivery {
    static double fee(double total, int kg, String city) {
        double fee = 4.50;
        if (!city.equals("Sofia")) {
            fee = fee + 2.00;
        }
        if (city.equals("Vidin")) {
            fee = fee + 3.00;
        }
        if (kg > 10) {
            fee = fee + (kg - 10) * 0.50;
        }
        if (total >= 50) {
            fee = 0;
        }
        return fee;
    }
}

Last month a colleague "tidied it up": if the order is free, why calculate anything, so the check moved to the top. They wrote total > 50 instead of total >= 50. For a whole week, orders of exactly €50 paid for delivery. Nobody noticed until the complaints came in.

Now you need to add a new city. Do you touch this method? If you don't, every new rule gets stuck on top and the code gets scarier and scarier. If you do, you might break something that works and never find out.

The idea

A net under the rope

A tightrope walker walks calmly because there is a net below. If they fall, the net catches them before anything bad happens.

For code, the net is tests. Before you change how the code is written, you write tests that say what it does today. Then you change it and run the tests. Green: the behaviour is the same. Red: something has changed, and you know exactly what.

Tests like these are called characterization tests: they don't check whether the code is right, they describe how it behaves. If the code does something odd today, the test records the odd part too. This is the safety net: no code gets touched until there are tests under it.

See it

Write down what it does today

You don't guess what the method ought to return. You ask it. A short program calls it with a few orders and prints the answers:

System.out.println("€20, 2 kg, Sofia → " + Delivery.fee(20, 2, "Sofia"));
System.out.println("€20, 2 kg, Plovdiv → " + Delivery.fee(20, 2, "Plovdiv"));
System.out.println("€20, 16 kg, Sofia → " + Delivery.fee(20, 16, "Sofia"));
System.out.println("€50, 2 kg, Plovdiv → " + Delivery.fee(50, 2, "Plovdiv"));
System.out.println("€60, 25 kg, Plovdiv → " + Delivery.fee(60, 25, "Plovdiv"));

Output

€20, 2 kg, Sofia → 4.5
€20, 2 kg, Plovdiv → 6.5
€20, 16 kg, Sofia → 7.5
€50, 2 kg, Plovdiv → 0.0
€60, 25 kg, Plovdiv → 0.0

The last line is a surprise. The parcel is 25 kg, and delivery is free: free delivery also drops the weight surcharge. Is that a bug? Maybe. But that is how it works today, customers rely on it, and whether to change it is not your call while you are tidying the code. You write it down as it is.

Each line of the probe becomes a test:

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;

class DeliveryTest {
    @Test
    void sofia() {
        assertEquals(4.5, Delivery.fee(20, 2, "Sofia"));
    }

    @Test
    void outsideSofia() {
        assertEquals(6.5, Delivery.fee(20, 2, "Plovdiv"));
    }

    @Test
    void heavy() {
        assertEquals(7.5, Delivery.fee(20, 16, "Sofia"));
    }

    @Test
    void freeFrom50() {
        assertEquals(0.0, Delivery.fee(50, 2, "Plovdiv"));
    }

    @Test
    void freeEvenWhenHeavy() {
        // That is how it works today: free delivery also drops the weight surcharge.
        assertEquals(0.0, Delivery.fee(60, 25, "Plovdiv"));
    }
}

Result

PASS  freeEvenWhenHeavy()
PASS  freeFrom50()
PASS  heavy()
PASS  outsideSofia()
PASS  sofia()
Tests: 5, passed: 5, failed: 0

Five tests, five green. The net is up.

More precisely, if you are curious

The tests here run the same way as in the testing lesson of the module "Programs in the real world". The checked program puts the class and the test in one file and runs JUnit 5 through the Launcher API, sorting the results by name. That way the output is the same on every run.

The values in the test are what Java printed. 4.5 and 0.0 are double, and the amounts here are multiples of 0.50, which are stored exactly, so assertEquals without a tolerance is enough.

See it

The harmless change under the net

Now the colleague makes the same change, but under a net. The check goes to the top:

class Delivery {
    static double fee(double total, int kg, String city) {
        if (total > 50) {
            return 0;
        }
        double fee = 4.50;
        if (!city.equals("Sofia")) {
            fee = fee + 2.00;
        }
        if (city.equals("Vidin")) {
            fee = fee + 3.00;
        }
        if (kg > 10) {
            fee = fee + (kg - 10) * 0.50;
        }
        return fee;
    }
}

Result

FAIL  freeFrom50(): expected: <0.0> but was: <6.5>
PASS  freeEvenWhenHeavy()
PASS  heavy()
PASS  outsideSofia()
PASS  sofia()
Tests: 5, passed: 4, failed: 1

One test is red, and it says exactly which case: an order of €50 outside Sofia. Expected 0, got €6.50. The bug is caught at the desk, not by a customer.

class DeliveryTest {
    @Test
    void sofia() {
        assertEquals(4.5, Delivery.fee(20, 2, "Sofia"));
    }

    @Test
    void outsideSofia() {
        assertEquals(6.5, Delivery.fee(20, 2, "Plovdiv"));
    }

    @Test
    void heavy() {
        assertEquals(7.5, Delivery.fee(20, 16, "Sofia"));
    }

    @Test
    void freeFrom50() {
        assertEquals(0.0, Delivery.fee(50, 2, "Plovdiv"));
    }

    @Test
    void freeEvenWhenHeavy() {
        // That is how it works today: free delivery also drops the weight surcharge.
        assertEquals(0.0, Delivery.fee(60, 25, "Plovdiv"));
    }
}

Step by step, in words

1. Five tests, one for each line of the probe. Each one calls Delivery.fee and compares with the answer it gives today.
2. Delivery.fee(20, 2, "Sofia") returns 4.5. The test pins it down.
3. Delivery.fee(20, 2, "Plovdiv") returns 6.5. The test pins it down.
4. Delivery.fee(20, 16, "Sofia") returns 7.5. The test pins it down.
5. Delivery.fee(50, 2, "Plovdiv") returns 0.0. The test pins it down.
6. Delivery.fee(60, 25, "Plovdiv") returns 0.0, although the parcel is 25 kg. Odd, but that is how it is today: the test keeps it, it doesn't fix it.
7. A colleague moves the free delivery check to the top and writes > instead of >=. The whole net runs again.
8. freeFrom50() fails: expected 0.0, but got 6.5. An order of exactly €50 now pays for delivery.
9. With >= all five tests are green. The change is safe: the net has checked it.

With >= everything is green:

class Delivery {
    static double fee(double total, int kg, String city) {
        if (total >= 50) {
            return 0;
        }
        double fee = 4.50;
        if (!city.equals("Sofia")) {
            fee = fee + 2.00;
        }
        if (city.equals("Vidin")) {
            fee = fee + 3.00;
        }
        if (kg > 10) {
            fee = fee + (kg - 10) * 0.50;
        }
        return fee;
    }
}

Result

PASS  freeEvenWhenHeavy()
PASS  freeFrom50()
PASS  heavy()
PASS  outsideSofia()
PASS  sofia()
Tests: 5, passed: 5, failed: 0

The change is the same one that went unnoticed for a whole week. The only difference is the net.

Try it

Predict the output

Someone changes the line for the weight surcharge and forgets to subtract the first 10 kg:

fee = fee + kg * 0.50;

The tests from the card "Write down what it does today" run again. What happens?

class Delivery {
    static double fee(double total, int kg, String city) {
        double fee = 4.50;
        if (!city.equals("Sofia")) {
            fee = fee + 2.00;
        }
        if (city.equals("Vidin")) {
            fee = fee + 3.00;
        }
        if (kg > 10) {
            fee = fee + kg * 0.50;
        }
        if (total >= 50) {
            fee = 0;
        }
        return fee;
    }
}

Answer

All five pass: the change is small

heavy() and freeEvenWhenHeavy() fail

Only heavy() fails

Check

Try it

What you write as expected

For €60, 25 kg and Plovdiv the probe printed: 0.0. You are sure the right delivery is €14. What do you write in the test before the change?

@Test
void freeEvenWhenHeavy() {
    assertEquals(, Delivery.fee(60, 25, "Plovdiv"));
}

Check

The idea

Small steps

The net helps most when you fall from low down. If you change twenty things at once and one test fails, you don't know which of the twenty is to blame. If you change one thing and the test fails, the culprit is obvious.

So the work has a rhythm: one small change, run the tests, green, the next one. Red means: undo the last change (Ctrl+Z, or git checkout on the file) and make it in even smaller pieces. That way the code is in a working state almost all the time.

Put the steps in order:

1. Run the tests again↑↓
2. Green: keep it (commit). Red: undo only the last change↑↓
3. Add tests for the path you are about to touch↑↓
4. Make one small change↑↓
5. Run the tests: all of them are green↑↓

Check

Try it

Find the bug

Someone wrote the test for €50 like this. Then the colleague makes the change with total > 50 from the card "The harmless change under the net":

class DeliveryTest {
    @Test
    void sofia() {
        assertEquals(4.5, Delivery.fee(20, 2, "Sofia"));
    }

    @Test
    void outsideSofia() {
        assertEquals(6.5, Delivery.fee(20, 2, "Plovdiv"));
    }

    @Test
    void heavy() {
        assertEquals(7.5, Delivery.fee(20, 16, "Sofia"));
    }

    @Test
    void freeFrom50() {
        System.out.println(Delivery.fee(50, 2, "Plovdiv"));
    }

    @Test
    void freeEvenWhenHeavy() {
        // That is how it works today: free delivery also drops the weight surcharge.
        assertEquals(0.0, Delivery.fee(60, 25, "Plovdiv"));
    }
}

Result

6.5
PASS  freeEvenWhenHeavy()
PASS  freeFrom50()
PASS  heavy()
PASS  outsideSofia()
PASS  sofia()
Tests: 5, passed: 5, failed: 0

The wrong amount is printed, and the tests are green. Which line is the bug on?

Pick the line with the bug.

1class DeliveryTest {2 @Test3 void sofia() {4 assertEquals(4.5, Delivery.fee(20, 2, "Sofia"));5 }67 @Test8 void outsideSofia() {9 assertEquals(6.5, Delivery.fee(20, 2, "Plovdiv"));10 }1112 @Test13 void heavy() {14 assertEquals(7.5, Delivery.fee(20, 16, "Sofia"));15 }1617 @Test18 void freeFrom50() {19 System.out.println(Delivery.fee(50, 2, "Plovdiv"));20 }2122 @Test23 void freeEvenWhenHeavy() {24 // That is how it works today: free delivery also drops the weight surcharge.25 assertEquals(0.0, Delivery.fee(60, 25, "Plovdiv"));26 }27}

Check

The idea

What the net doesn't catch

The net catches only where it is stretched. There is no test for Vidin. Someone "tidies up" the cities with else if:

class Delivery {
    static double fee(double total, int kg, String city) {
        double fee = 4.50;
        if (city.equals("Vidin")) {
            fee = fee + 3.00;
        } else if (!city.equals("Sofia")) {
            fee = fee + 2.00;
        }
        if (kg > 10) {
            fee = fee + (kg - 10) * 0.50;
        }
        if (total >= 50) {
            fee = 0;
        }
        return fee;
    }
}

The tests

PASS  freeEvenWhenHeavy()
PASS  freeFrom50()
PASS  heavy()
PASS  outsideSofia()
PASS  sofia()
Tests: 5, passed: 5, failed: 0

Vidin before and after

Vidin before: 9.5
Vidin after: 7.5

The tests are green, and delivery to Vidin got €2 cheaper: before, Vidin paid both the €2 for outside Sofia and €3 on top. No test goes down this path, so no test fails.

Before you touch code, check which paths have tests and add some for the ones you will touch. That is what coverage shows: which lines ran while the tests were running.

In IntelliJ IDEA you run the tests with "Run with Coverage", and lines without a test light up red. In a Maven or Gradle project, JaCoCo gives you the same.There is one thing, though, that the compiler protects even without a test. If you rename fee by hand and forget it in the Vidin branch, the program doesn't compile: fee no longer exists. The bug is found before there is a test.

More precisely, if you are curious

100% coverage doesn't mean every case is checked. The line with total >= 50 is "covered" by a test with €60 too, but exactly €50 is a separate case, and only the €50 test catches >. Coverage tells you where the net has holes. How dense it is where it exists depends on the cases you picked: the boundaries (€50, 10 kg), every branch of an if, and the odd values from the probe.

The coverage commands are not run here: the site's checks run without a network and without external packages.

Exercise

A net for Vidin

Solve the task in your editor.

A net for Vidin. Add two tests to the tests from the card "Write down what it does today": an order of €20, 2 kg to Vidin and an order of €50, 2 kg to Vidin. Run the probe first to see what the code returns today, and pin that down.

Result

PASS  freeEvenWhenHeavy()
PASS  freeFrom50()
PASS  heavy()
PASS  outsideSofia()
PASS  sofia()
PASS  vidin()
PASS  vidinFreeFrom50()
Tests: 7, passed: 7, failed: 0

Then make the change with else if from the card "What the net doesn't catch". Now it has to fail:

Result

FAIL  vidin(): expected: <9.5> but was: <7.5>
PASS  freeEvenWhenHeavy()
PASS  freeFrom50()
PASS  heavy()
PASS  outsideSofia()
PASS  sofia()
PASS  vidinFreeFrom50()
Tests: 7, passed: 6, failed: 1

More tasks

• The weight boundary. Add a test for exactly 10 kg and for 11 kg. Then change kg > 10 to kg >= 10. Does anything fail? Why?
• The new city. Add Varna with €1 on top. First a test for Varna that fails, then the rule, then all the tests again.
• A probe for everything. Write the probe as a loop over a table of cases (amount, weight, city), so that a new case is a new row in the table.

Test

Check yourself

1. The probe shows that an order of €60 with a 25 kg parcel is delivered for free. You think this is a bug. What do you write in the test before the change?

The value that ought to be rightThe value the function returns today, with a note that it looks oddNothing: you don't write a test for a case like that

2. Five tests pass. You rearrange the function and one fails. What do you do?

Change the expected value in the test so that it passesDelete the test, since it is in the wayUndo the last change and make it in smaller steps

3. Why are the steps small, with the tests run after each one?

To have more commitsWhen a test fails, you know the last small change is to blameBecause tests are fast only for small code

4. All tests are green. Does that mean the change didn't change anything?

Yes, if there are more than tenNo: tests protect only the cases they check. A path without a test may have changedYes, since they are green

5. A test calls the function and prints the result without comparing it. What does it protect?

Only the case it printsNothing: it passes with any result, unless the program stops with an errorEverything it prints

Check

Start the lesson over

Related lessons

• Builds on: Tests with JUnit · Names in code · Functions: one job, few parameters, no hidden side effects · Code smells: long method, large class, duplication, primitives instead of types, long if chains
• Continues in: Refactoring in small steps · Exercise: a messy class, cleaned up step by step · Dependencies: other people’s code that yours runs
• Tasks on this topic: Clean code and refactoring