Code coverage

In short: Statement coverage asks whether every line of code has run. Branch coverage asks whether every decision has gone both ways. For the delivery function the first takes 2 tests, the second 3. 100 % branch coverage includes 100 % statement coverage, but not the other way round. And no coverage says the result was checked or that the boundary is in the right place.

Problem

100 % coverage, a wrong fee

The developer runs the delivery tests with a tool that counts which lines ran. The tests are two: 0 € and 60 €. The report says: 100 % of the lines.

The next day customers with a 30 € cart pay a 4.99 € delivery fee instead of 4.90 €. The line with the fee did run. No test saw what was in it.

The idea

The white box

Until now the tests came from the rule: that is the black box. White-box techniques take the tests from the code: which lines it has and which paths the program can take.

You do not need to be able to write code for this lesson. You need to be able to read the delivery function. The language is your choice, in the tabs:

Java

static String delivery(double total) {
    if (total <= 0) {
        return "refused";
    }
    double fee = 4.90;
    if (total >= 50) {
        fee = 0;
    }
    return String.format("%.2f €", fee);
}

C#

static string Delivery(double total)
{
    if (total <= 0)
    {
        return "refused";
    }
    double fee = 4.90;
    if (total >= 50)
    {
        fee = 0;
    }
    return $"{fee:F2} €";
}

Python

def delivery(total):
    if total <= 0:
        return "refused"
    fee = 4.90
    if total >= 50:
        fee = 0
    return f"{fee:.2f} €"

JavaScript

function delivery(total) {
  if (total <= 0) {
    return "refused";
  }
  let fee = 4.9;
  if (total >= 50) {
    fee = 0;
  }
  return `${fee.toFixed(2)} €`;
}

The function has two decisions (places where the program chooses a path): "the total is 0 or less" and "the total is 50 or more". The second is an "if" without an "else": if it is false, the program simply skips the line that sets the fee to zero.

The idea

Statement coverage

Statement coverage asks: how many of the lines that do something has at least one test run?

Here are the two tests from the start, as calls of the function, with their output:

Java

System.out.println("0 € → " + delivery(0));
System.out.println("60 € → " + delivery(60));

Output

0 € → refused
60 € → 0.00 €

C#

Console.WriteLine("0 € → " + Delivery(0));
Console.WriteLine("60 € → " + Delivery(60));

Output

0 € → refused
60 € → 0.00 €

Python

print("0 € →", delivery(0))
print("60 € →", delivery(60))

Output

0 € → refused
60 € → 0.00 €

JavaScript

console.log("0 € → " + delivery(0));
console.log("60 € → " + delivery(60));

Output

0 € → refused
60 € → 0.00 €

• 0 € goes through the first check and returns "refused".
• 60 € goes through everything else: the fee becomes 4.90, then the "from 50" check makes it 0, then the return.

Every line has run. Statement coverage is 100 %, with two tests.

The idea

Branch coverage

Branch coverage asks something else: has every decision gone both ways?

There are four branches here: the first check true and false, the second true and false. 0 € takes the first as true. 60 € takes the first as false and the second as true. Nobody takes the second as false: that is the empty path of an "if" without an "else". Coverage is 3 of 4, that is 75 %.

A third test, with paid delivery, closes the gap:

Java

System.out.println("0 € → " + delivery(0));
System.out.println("30 € → " + delivery(30));
System.out.println("60 € → " + delivery(60));

Output

0 € → refused
30 € → 4.90 €
60 € → 0.00 €

C#

Console.WriteLine("0 € → " + Delivery(0));
Console.WriteLine("30 € → " + Delivery(30));
Console.WriteLine("60 € → " + Delivery(60));

Output

0 € → refused
30 € → 4.90 €
60 € → 0.00 €

Python

print("0 € →", delivery(0))
print("30 € →", delivery(30))
print("60 € →", delivery(60))

Output

0 € → refused
30 € → 4.90 €
60 € → 0.00 €

JavaScript

console.log("0 € → " + delivery(0));
console.log("30 € → " + delivery(30));
console.log("60 € → " + delivery(60));

Output

0 € → refused
30 € → 4.90 €
60 € → 0.00 €

Three tests, 100 % of the branches. And the answer to the lesson's question: 100 % branch coverage of the delivery function takes 3 tests.

More precisely, if you are curious

ISTQB CTFL v4.0.1 describes both measures in section 4.3: statement testing and branch testing. Coverage is the number of executed statements (or taken branches) over all of them, as a percentage. The syllabus also states how they relate: 100 % branch coverage includes 100 % statement coverage, but not the other way round.

There are stricter measures too: condition coverage (every condition inside a decision has been seen both true and false) and MC/DC (every condition has shown that it alone can change the decision). MC/DC is required for the most critical software in aviation. They are not in CTFL v4.0.

Coverage is measured with a tool: JaCoCo for Java, coverlet for C#, coverage.py for Python, c8 or the one built into Node for JavaScript. The tests are run, and the tool marks which lines and branches were taken.

Try it

How many tests?

1. A test with a cart of 0 €. Which parts of the function does it run?

Nothing: the total is 0Only the first check and returning "refused"Every line

2. The tests are 0 € and 60 €. What is the branch coverage?

100 %3 of 4 (75 %): the check "from 50" has never been false50 %

3. Which test brings branch coverage to 100 %?

A cart of 100 €A cart between 0.01 and 49.99 €, for example 30 €A second test with 0 €

Check

The idea

Why 100 % statements missed the fee

Here is the function from the start, with the 4.99 fee:

Java

static String delivery(double total) {
    if (total <= 0) {
        return "refused";
    }
    double fee = 4.99;
    if (total >= 50) {
        fee = 0;
    }
    return String.format("%.2f €", fee);
}

C#

static string Delivery(double total)
{
    if (total <= 0)
    {
        return "refused";
    }
    double fee = 4.99;
    if (total >= 50)
    {
        fee = 0;
    }
    return $"{fee:F2} €";
}

Python

def delivery(total):
    if total <= 0:
        return "refused"
    fee = 4.99
    if total >= 50:
        fee = 0
    return f"{fee:.2f} €"

JavaScript

function delivery(total) {
  if (total <= 0) {
    return "refused";
  }
  let fee = 4.99;
  if (total >= 50) {
    fee = 0;
  }
  return `${fee.toFixed(2)} €`;
}

With the two statement tests the output is the same as for the correct function:

Java

System.out.println("0 € → " + delivery(0));
System.out.println("60 € → " + delivery(60));

Output

0 € → refused
60 € → 0.00 €

C#

Console.WriteLine("0 € → " + Delivery(0));
Console.WriteLine("60 € → " + Delivery(60));

Output

0 € → refused
60 € → 0.00 €

Python

print("0 € →", delivery(0))
print("60 € →", delivery(60))

Output

0 € → refused
60 € → 0.00 €

JavaScript

console.log("0 € → " + delivery(0));
console.log("60 € → " + delivery(60));

Output

0 € → refused
60 € → 0.00 €

At 60 € the line with the fee runs, but right after it the fee becomes 0. The wrong value never reaches the output. Run does not mean checked.

The third test, the one for full branch coverage, takes the empty path and shows the fee:

Java

System.out.println("0 € → " + delivery(0));
System.out.println("30 € → " + delivery(30));
System.out.println("60 € → " + delivery(60));

Output

0 € → refused
30 € → 4.99 €
60 € → 0.00 €

C#

Console.WriteLine("0 € → " + Delivery(0));
Console.WriteLine("30 € → " + Delivery(30));
Console.WriteLine("60 € → " + Delivery(60));

Output

0 € → refused
30 € → 4.99 €
60 € → 0.00 €

Python

print("0 € →", delivery(0))
print("30 € →", delivery(30))
print("60 € →", delivery(60))

Output

0 € → refused
30 € → 4.99 €
60 € → 0.00 €

JavaScript

console.log("0 € → " + delivery(0));
console.log("30 € → " + delivery(30));
console.log("60 € → " + delivery(60));

Output

0 € → refused
30 € → 4.99 €
60 € → 0.00 €

That is why branch coverage is stronger. The empty path of an "if" is exactly where a value from the line above reaches the customer.

The idea

What coverage does not say

Here is the cart limit check: adding is refused if it is less than 1 item, or if the cart would hold more than 10.

Java

static boolean canAdd(int inCart, int add) {
    if (add < 1 || inCart + add > 10) {
        return false;
    }
    return true;
}

C#

static bool CanAdd(int inCart, int add)
{
    if (add < 1 || inCart + add > 10)
    {
        return false;
    }
    return true;
}

Python

def can_add(in_cart, add):
    if add < 1 or in_cart + add > 10:
        return False
    return True

JavaScript

function canAdd(inCart, add) {
  if (add < 1 || inCart + add > 10) {
    return false;
  }
  return true;
}

Two calls give 100 % of the branches: (5, 0) goes through "refused", (2, 3) through "accepted".

Java

System.out.println(canAdd(5, 0));
System.out.println(canAdd(2, 3));

Output

false
true

C#

Console.WriteLine(CanAdd(5, 0));
Console.WriteLine(CanAdd(2, 3));

Output

False
True

Python

print(can_add(5, 0))
print(can_add(2, 3))

Output

False
True

JavaScript

console.log(canAdd(5, 0));
console.log(canAdd(2, 3));

Output

false
true

Now change "> 10" to "> 11". The same two tests pass, coverage is still 100 %, and the 11th item gets into the cart. Coverage measures what ran, not how close to the boundary. The test (10, 1) from the lesson on boundary values catches the bug.

And something no coverage can see: missing code. If the check for an empty cart was never written, there is no line left uncovered. 100 % coverage of code that does not do what is needed is still 100 %. That is why the white box does not replace the black box. Coverage says where there are no tests, not that the tests are good.

Exercise

The coverage of the limit

1. Two conditions, one decision. The questions are about the limit check above.

1. How many tests does 100 % branch coverage of the limit check take?

2: one where the condition is true and one where it is false4: one per combination of the two conditions1

2. In the code "> 10" is changed to "> 11". Do the tests (5, 0) and (2, 3) give 100 % branch coverage?

No: the coverage dropsYes, and they failYes, and they still pass: the bug is at the boundary, which no test reaches

3. With the tests (5, 0) and (2, 3), is the part "the cart plus what is added is over 10" evaluated at all?

No, neverFor both it is trueOnly for (2, 3), and there it is false; no test makes it true

Check

Test

Check yourself

1. Where do the tests come from in white-box techniques?

From the requirementsFrom the tester's experienceFrom the structure of the code

2. Which is true?

100 % statement coverage gives 100 % branch coverageThe two are the same100 % branch coverage also gives 100 % statement coverage

3. The tests give 100 % statement coverage, and a fee of 4.99 € instead of 4.90 € slips through. Why?

The coverage was calculated wronglyBecause the code is in JavaThe line with the fee runs, but its value is changed before anyone sees it

4. What can no code coverage say?

Which branches were takenWhich lines ranThat the code does what it should, and that nothing missing from the code was needed

Check

Start the lesson over

Related lessons

• Builds on: Why early · Equivalence partitions · Boundary values · Decision tables
• Continues in: Testing from experience
• Tasks on this topic: Test design techniques