Tests as a safety net

Code in:JavaC#PythonJavaScript

In short: Characterization tests pin down what old code does today, so they catch any accidental change in behaviour while you clean it up.

Problem

The harmless change

The shop works out delivery with the method Delivery.Fee. It was written three years ago and has no tests. Delivery is €4.50, outside Sofia it is €2 more, to Vidin another €3, and above 10 kg you pay €0.50 for every extra kilogram. From €50 up it is free.

public static class Delivery
{
    public static decimal Fee(decimal total, int kg, string city)
    {
        decimal fee = 4.50m;
        if (city != "Sofia")
        {
            fee = fee + 2.00m;
        }
        if (city == "Vidin")
        {
            fee = fee + 3.00m;
        }
        if (kg > 10)
        {
            fee = fee + (kg - 10) * 0.50m;
        }
        if (total >= 50)
        {
            fee = 0;
        }
        return fee;
    }
}

Last month a colleague "tidied it up": if the order is free, why calculate anything, so the check moved to the top. They wrote total > 50 instead of total >= 50. For a whole week, orders of exactly €50 paid for delivery. Nobody noticed until the complaints came in.

Now you need to add a new city. Do you touch this method? If you don't, every new rule gets stuck on top and the code gets scarier and scarier. If you do, you might break something that works and never find out.

The idea

A net under the rope

A tightrope walker walks calmly because there is a net below. If they fall, the net catches them before anything bad happens.

For code, the net is tests. Before you change how the code is written, you write tests that say what it does today. Then you change it and run the tests. Green: the behaviour is the same. Red: something has changed, and you know exactly what.

Tests like these are called characterization tests: they don't check whether the code is right, they describe how it behaves. If the code does something odd today, the test records the odd part too. This is the safety net: no code gets touched until there are tests under it.

See it

Write down what it does today

You don't guess what the method ought to return. You ask it. A short program calls it with a few orders and prints the answers:

Console.WriteLine("€20, 2 kg, Sofia → " + Delivery.Fee(20, 2, "Sofia"));
Console.WriteLine("€20, 2 kg, Plovdiv → " + Delivery.Fee(20, 2, "Plovdiv"));
Console.WriteLine("€20, 16 kg, Sofia → " + Delivery.Fee(20, 16, "Sofia"));
Console.WriteLine("€50, 2 kg, Plovdiv → " + Delivery.Fee(50, 2, "Plovdiv"));
Console.WriteLine("€60, 25 kg, Plovdiv → " + Delivery.Fee(60, 25, "Plovdiv"));

Output

€20, 2 kg, Sofia → 4.50
€20, 2 kg, Plovdiv → 6.50
€20, 16 kg, Sofia → 7.50
€50, 2 kg, Plovdiv → 0
€60, 25 kg, Plovdiv → 0

The last line is a surprise. The parcel is 25 kg, and delivery is free: free delivery also drops the weight surcharge. Is that a bug? Maybe. But that is how it works today, customers rely on it, and whether to change it is not your call while you are tidying the code. You write it down as it is.

Each line of the probe becomes a test:

using Xunit;

public class DeliveryTest
{
    [Fact]
    public void Sofia()
    {
        Assert.Equal(4.50m, Delivery.Fee(20, 2, "Sofia"));
    }

    [Fact]
    public void OutsideSofia()
    {
        Assert.Equal(6.50m, Delivery.Fee(20, 2, "Plovdiv"));
    }

    [Fact]
    public void Heavy()
    {
        Assert.Equal(7.50m, Delivery.Fee(20, 16, "Sofia"));
    }

    [Fact]
    public void FreeFrom50()
    {
        Assert.Equal(0m, Delivery.Fee(50, 2, "Plovdiv"));
    }

    [Fact]
    public void FreeEvenWhenHeavy()
    {
        // That is how it works today: free delivery also drops the weight surcharge.
        Assert.Equal(0m, Delivery.Fee(60, 25, "Plovdiv"));
    }
}

Result

PASS  FreeEvenWhenHeavy
PASS  FreeFrom50
PASS  Heavy
PASS  OutsideSofia
PASS  Sofia
Tests: 5, passed: 5, failed: 0

Five tests, five green. The net is up.

More precisely, if you are curious

The tests here run the same way as in the testing lesson of the module "Programs in the real world". The checked program is a project with Delivery.cs, DeliveryTest.cs and Runner.cs, which runs xUnit through AssemblyRunner and sorts the results by name. That way the output is the same on every run.

The values in the test are what C# printed. Money is decimal, so 4.50m is exactly 4.50, and 0m is zero.

See it

The harmless change under the net

Now the colleague makes the same change, but under a net. The check goes to the top:

public static class Delivery
{
    public static decimal Fee(decimal total, int kg, string city)
    {
        if (total > 50)
        {
            return 0;
        }
        decimal fee = 4.50m;
        if (city != "Sofia")
        {
            fee = fee + 2.00m;
        }
        if (city == "Vidin")
        {
            fee = fee + 3.00m;
        }
        if (kg > 10)
        {
            fee = fee + (kg - 10) * 0.50m;
        }
        return fee;
    }
}

Result

FAIL  FreeFrom50
      Assert.Equal() Failure: Values differ
      Expected: 0
      Actual:   6.50
PASS  FreeEvenWhenHeavy
PASS  Heavy
PASS  OutsideSofia
PASS  Sofia
Tests: 5, passed: 4, failed: 1

One test is red, and it says exactly which case: an order of €50 outside Sofia. Expected 0, got €6.50. The bug is caught at the desk, not by a customer.

public class DeliveryTest
{
    [Fact]
    public void Sofia()
    {
        Assert.Equal(4.50m, Delivery.Fee(20, 2, "Sofia"));
    }

    [Fact]
    public void OutsideSofia()
    {
        Assert.Equal(6.50m, Delivery.Fee(20, 2, "Plovdiv"));
    }

    [Fact]
    public void Heavy()
    {
        Assert.Equal(7.50m, Delivery.Fee(20, 16, "Sofia"));
    }

    [Fact]
    public void FreeFrom50()
    {
        Assert.Equal(0m, Delivery.Fee(50, 2, "Plovdiv"));
    }

    [Fact]
    public void FreeEvenWhenHeavy()
    {
        // That is how it works today: free delivery also drops the weight surcharge.
        Assert.Equal(0m, Delivery.Fee(60, 25, "Plovdiv"));
    }
}

Step by step, in words

1. Five tests, one for each line of the probe. Each one calls Delivery.Fee and compares with the answer it gives today.
2. Delivery.Fee(20, 2, "Sofia") returns 4.50. The test pins it down.
3. Delivery.Fee(20, 2, "Plovdiv") returns 6.50. The test pins it down.
4. Delivery.Fee(20, 16, "Sofia") returns 7.50. The test pins it down.
5. Delivery.Fee(50, 2, "Plovdiv") returns 0. The test pins it down.
6. Delivery.Fee(60, 25, "Plovdiv") returns 0, although the parcel is 25 kg. Odd, but that is how it is today: the test keeps it, it doesn't fix it.
7. A colleague moves the free delivery check to the top and writes > instead of >=. The whole net runs again.
8. FreeFrom50 fails: expected 0, but got 6.50. An order of exactly €50 now pays for delivery.
9. With >= all five tests are green. The change is safe: the net has checked it.

With >= everything is green:

public static class Delivery
{
    public static decimal Fee(decimal total, int kg, string city)
    {
        if (total >= 50)
        {
            return 0;
        }
        decimal fee = 4.50m;
        if (city != "Sofia")
        {
            fee = fee + 2.00m;
        }
        if (city == "Vidin")
        {
            fee = fee + 3.00m;
        }
        if (kg > 10)
        {
            fee = fee + (kg - 10) * 0.50m;
        }
        return fee;
    }
}

Result

PASS  FreeEvenWhenHeavy
PASS  FreeFrom50
PASS  Heavy
PASS  OutsideSofia
PASS  Sofia
Tests: 5, passed: 5, failed: 0

The change is the same one that went unnoticed for a whole week. The only difference is the net.

Try it

Predict the output

Someone changes the line for the weight surcharge and forgets to subtract the first 10 kg:

fee = fee + kg * 0.50m;

The tests from the card "Write down what it does today" run again. What happens?

public static class Delivery
{
    public static decimal Fee(decimal total, int kg, string city)
    {
        decimal fee = 4.50m;
        if (city != "Sofia")
        {
            fee = fee + 2.00m;
        }
        if (city == "Vidin")
        {
            fee = fee + 3.00m;
        }
        if (kg > 10)
        {
            fee = fee + kg * 0.50m;
        }
        if (total >= 50)
        {
            fee = 0;
        }
        return fee;
    }
}

Answer

All five pass: the change is small

Heavy and FreeEvenWhenHeavy fail

Only Heavy fails

Check

Try it

What you write as expected

For €60, 25 kg and Plovdiv the probe printed: 0. You are sure the right delivery is €14. What do you write in the test before the change?

[Fact]
public void FreeEvenWhenHeavy()
{
    Assert.Equal(, Delivery.Fee(60, 25, "Plovdiv"));
}

Check

The idea

Small steps

The net helps most when you fall from low down. If you change twenty things at once and one test fails, you don't know which of the twenty is to blame. If you change one thing and the test fails, the culprit is obvious.

So the work has a rhythm: one small change, run the tests, green, the next one. Red means: undo the last change (Ctrl+Z, or git checkout on the file) and make it in even smaller pieces. That way the code is in a working state almost all the time.

Put the steps in order:

1. Run the tests again↑↓
2. Green: keep it (commit). Red: undo only the last change↑↓
3. Add tests for the path you are about to touch↑↓
4. Make one small change↑↓
5. Run the tests: all of them are green↑↓

Check

Try it

Find the bug

Someone wrote the test for €50 like this. Then the colleague makes the change with total > 50 from the card "The harmless change under the net":

public class DeliveryTest
{
    [Fact]
    public void Sofia()
    {
        Assert.Equal(4.50m, Delivery.Fee(20, 2, "Sofia"));
    }

    [Fact]
    public void OutsideSofia()
    {
        Assert.Equal(6.50m, Delivery.Fee(20, 2, "Plovdiv"));
    }

    [Fact]
    public void Heavy()
    {
        Assert.Equal(7.50m, Delivery.Fee(20, 16, "Sofia"));
    }

    [Fact]
    public void FreeFrom50()
    {
        Console.WriteLine(Delivery.Fee(50, 2, "Plovdiv"));
    }

    [Fact]
    public void FreeEvenWhenHeavy()
    {
        // That is how it works today: free delivery also drops the weight surcharge.
        Assert.Equal(0m, Delivery.Fee(60, 25, "Plovdiv"));
    }
}

Result

6.50
PASS  FreeEvenWhenHeavy
PASS  FreeFrom50
PASS  Heavy
PASS  OutsideSofia
PASS  Sofia
Tests: 5, passed: 5, failed: 0

The wrong amount is printed, and the tests are green. Which line is the bug on?

Pick the line with the bug.

1public class DeliveryTest2{3 [Fact]4 public void Sofia()5 {6 Assert.Equal(4.50m, Delivery.Fee(20, 2, "Sofia"));7 }89 [Fact]10 public void OutsideSofia()11 {12 Assert.Equal(6.50m, Delivery.Fee(20, 2, "Plovdiv"));13 }1415 [Fact]16 public void Heavy()17 {18 Assert.Equal(7.50m, Delivery.Fee(20, 16, "Sofia"));19 }2021 [Fact]22 public void FreeFrom50()23 {24 Console.WriteLine(Delivery.Fee(50, 2, "Plovdiv"));25 }2627 [Fact]28 public void FreeEvenWhenHeavy()29 {30 // That is how it works today: free delivery also drops the weight surcharge.31 Assert.Equal(0m, Delivery.Fee(60, 25, "Plovdiv"));32 }33}

Check

The idea

What the net doesn't catch

The net catches only where it is stretched. There is no test for Vidin. Someone "tidies up" the cities with else if:

public static class Delivery
{
    public static decimal Fee(decimal total, int kg, string city)
    {
        decimal fee = 4.50m;
        if (city == "Vidin")
        {
            fee = fee + 3.00m;
        }
        else if (city != "Sofia")
        {
            fee = fee + 2.00m;
        }
        if (kg > 10)
        {
            fee = fee + (kg - 10) * 0.50m;
        }
        if (total >= 50)
        {
            fee = 0;
        }
        return fee;
    }
}

The tests

PASS  FreeEvenWhenHeavy
PASS  FreeFrom50
PASS  Heavy
PASS  OutsideSofia
PASS  Sofia
Tests: 5, passed: 5, failed: 0

Vidin before and after

Vidin before: 9.50
Vidin after: 7.50

The tests are green, and delivery to Vidin got €2 cheaper: before, Vidin paid both the €2 for outside Sofia and €3 on top. No test goes down this path, so no test fails.

Before you touch code, check which paths have tests and add some for the ones you will touch. That is what coverage shows: which lines ran while the tests were running.

The dotnet new xunit template already has coverlet.collector: dotnet test --collect:"XPlat Code Coverage" writes a report that ReportGenerator turns into a page. Rider and Visual Studio Enterprise show coverage in the editor.There is one thing, though, that the compiler protects even without a test. If you rename fee by hand and forget it in the Vidin branch, the program doesn't compile: fee no longer exists. The bug is found before there is a test.

More precisely, if you are curious

100% coverage doesn't mean every case is checked. The line with total >= 50 is "covered" by a test with €60 too, but exactly €50 is a separate case, and only the €50 test catches >. Coverage tells you where the net has holes. How dense it is where it exists depends on the cases you picked: the boundaries (€50, 10 kg), every branch of an if, and the odd values from the probe.

The coverage commands are not run here: the site's checks run without a network and without external packages.

Exercise

A net for Vidin

Solve the task in your editor.

A net for Vidin. Add two tests to the tests from the card "Write down what it does today": an order of €20, 2 kg to Vidin and an order of €50, 2 kg to Vidin. Run the probe first to see what the code returns today, and pin that down.

Result

PASS  FreeEvenWhenHeavy
PASS  FreeFrom50
PASS  Heavy
PASS  OutsideSofia
PASS  Sofia
PASS  Vidin
PASS  VidinFreeFrom50
Tests: 7, passed: 7, failed: 0

Then make the change with else if from the card "What the net doesn't catch". Now it has to fail:

Result

FAIL  Vidin
      Assert.Equal() Failure: Values differ
      Expected: 9.50
      Actual:   7.50
PASS  FreeEvenWhenHeavy
PASS  FreeFrom50
PASS  Heavy
PASS  OutsideSofia
PASS  Sofia
PASS  VidinFreeFrom50
Tests: 7, passed: 6, failed: 1

More tasks

• The weight boundary. Add a test for exactly 10 kg and for 11 kg. Then change kg > 10 to kg >= 10. Does anything fail? Why?
• The new city. Add Varna with €1 on top. First a test for Varna that fails, then the rule, then all the tests again.
• A probe for everything. Write the probe as a loop over a table of cases (amount, weight, city), so that a new case is a new row in the table.

Test

Check yourself

1. The probe shows that an order of €60 with a 25 kg parcel is delivered for free. You think this is a bug. What do you write in the test before the change?

The value that ought to be rightThe value the function returns today, with a note that it looks oddNothing: you don't write a test for a case like that

2. Five tests pass. You rearrange the function and one fails. What do you do?

Change the expected value in the test so that it passesDelete the test, since it is in the wayUndo the last change and make it in smaller steps

3. Why are the steps small, with the tests run after each one?

To have more commitsWhen a test fails, you know the last small change is to blameBecause tests are fast only for small code

4. All tests are green. Does that mean the change didn't change anything?

Yes, if there are more than tenNo: tests protect only the cases they check. A path without a test may have changedYes, since they are green

5. A test calls the function and prints the result without comparing it. What does it protect?

Only the case it printsNothing: it passes with any result, unless the program stops with an errorEverything it prints

Check

Start the lesson over

Related lessons

• Builds on: Tests with xUnit · Names in code · Functions: one job, few parameters, no hidden side effects · Code smells: long method, large class, duplication, primitives instead of types, long if chains
• Continues in: Refactoring in small steps · Exercise: a messy class, cleaned up step by step · Dependencies: other people’s code that yours runs
• Tasks on this topic: Clean code and refactoring