What coverage does not tell you
In short: Requirements coverage counts how many requirements have at least one test. It does not say whether the test catches the bug, whether it checks the whole requirement, what happens when two rules meet, or what has not been written down at all. That is why next to the number you always ask "what has not been checked, and why".
Problem
A hundred per cent, and still wrong
The report before the autumn campaign goes live is green: all requirements are covered, all tests pass.
A week later support gets a complaint. A cart of 52.00 € with AUTUMN10, 46.80 € after the discount. The cart says "Delivery: free", but at payment the customer is charged 4.90 €. The cart counts the threshold before the discount, and payment after it.
Delivery has tests. The promo code has tests. But no test runs the two rules together. And when QA asks which answer is right, it turns out nobody has decided: is the 50 € threshold counted before the discount or after it?
The idea
What the number counts
Requirements coverage counts the requirements that have at least one test. Four of four: 100 %. Nothing more.
The number does not say:
- Whether the test catches the bug. A test at 120.00 € covers "free from 50.00 €", but it does not catch "over 50 €". The tick is real, the test is weak.
- Whether it checks the whole requirement. "The code gives 10 % off and does not apply to discounted products" is two rules in one sentence. One test for the 10 %, and the row is covered.
- What happens between requirements. The matrix has a row per rule, not per meeting of two rules. The cart of 52.00 € with a code is in no row.
- What has not been written down. There are rows only for the requirements in the list. A requirement nobody wrote down does not make an empty row. It is simply missing.
These are the limits of coverage. Coverage is useful. Zero is an alarm, and an empty row is a real hole. But 100 % only says there are no empty rows.
More precisely, if you are curious
Coverage is always measured against something: requirements, input classes, boundaries, transitions, branches in the code. You will see them in the module "Test design techniques". 100 % by one criterion means that all its items have been exercised at least once. It does not mean there are no defects: testing shows that defects are present, not that they are absent. That is the first of the seven principles in ISTQB CTFL v4.0.1 (section 1.3).
ISTQB CTFL v4.0.1 (section 1.4.4) names coverage evaluation as one of the benefits of traceability. Coverage answers the question "is there a test for every requirement". How well the test checks is a different question, with its own measures.
Try it
Where does the number lie?
Try it
A hundred per cent on the discounts
Mark the links, then the two pairs of tests that check the same thing. See what the coverage is and what still has not been checked.
Tests
- T-1: A cart of 30.00 €: delivery is 4.90 €.
- T-2: A cart of 120.00 €: delivery is free.
- T-3: A cart of 25.00 €: delivery is 4.90 €.
- T-4: A T-shirt for 20.00 € with AUTUMN10: 18.00 €.
- T-5: A loyal customer, discounted jeans for 40.00 €: 38.00 €.
- T-6: A loyal customer with AUTUMN10, a T-shirt for 20.00 €: 18.00 €, not 17.00 €.
- T-7: A T-shirt for 30.00 € with AUTUMN10: 27.00 €.
Press the cell where the test checks the requirement; press again to remove the link. The arrow keys move between the cells, Space or Enter marks.
| Requirements | T-1: A cart of 30.00 €: delivery is 4.90 €. | T-2: A cart of 120.00 €: delivery is free. | T-3: A cart of 25.00 €: delivery is 4.90 €. | T-4: A T-shirt for 20.00 € with AUTUMN10: 18.00 €. | T-5: A loyal customer, discounted jeans for 40.00 €: 38.00 €. | T-6: A loyal customer with AUTUMN10, a T-shirt for 20.00 €: 18.00 €, not 17.00 €. | T-7: A T-shirt for 30.00 € with AUTUMN10: 27.00 €. | Covered by |
|---|---|---|---|---|---|---|---|---|
| R-1 Delivery is free from 50.00 € inclusive, otherwise it is 4.90 €. | 0 | |||||||
| R-2 A promo code gives 10 % off and does not apply to discounted products. | 0 | |||||||
| R-3 A loyal customer gets 5 % off, on discounted products too. | 0 | |||||||
| R-4 Discounts do not add up: the bigger one applies. | 0 | |||||||
| Checks | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
The reference matrix
- R-1: T-1, T-2, T-3
- R-2: T-4, T-6, T-7
- R-3: T-5
- R-4: T-6
Redundant: T-1 / T-3 (30 € and 25 € are in the same group: under 50 €.); T-4 / T-7 (a T-shirt for 20 € and for 30 € with the code are the same case: a product not on sale, with 10 % off.)
Every requirement is covered, every test belongs to a requirement. And still:
- No test is at 50.00 €. If the threshold becomes "over 50 €", everything stays green.
- R-2 is covered by three tests, but all of them are about the 10 %. The code on a discounted product has not been checked.
- T-6 is not linked to R-3: 18.00 € comes out even if loyalty does not work at all. If you link it, the coverage does not change, but the matrix starts to lie.
- The cart of 52.00 € with a code is nowhere, because the rule for it has not been decided yet.
Two of the seven tests repeat others. Without them the coverage is still 100 %, and the holes are the same.
The idea
The bug is between them
Most rules in the shop are simple on their own. Delivery is one comparison. The promo code is 10 %. It gets hard when they meet:
- the delivery threshold and the discount: before or after?
- the code and a return: if the paid order with the code is returned, can the code be used again?
- the code and loyalty on a discounted product: the code does not apply, loyalty does, so 5 %. The answer follows from the rules, but there is no test for it.
The first two questions have no row in the matrix, because they have no requirement. The third has an answer, but it sits between two rows. This is also the most valuable thing you can do with the matrix: look at pairs of rows and ask "and when both apply?". Sometimes the answer is a new test. More often it is an open question for the owner, and then a new requirement.
So the empty place in the matrix is not only an empty row. Sometimes it is between two full ones.
The idea
When the number becomes the target
If management asks for "100 % coverage by Friday" and looks only at that, they will get it. The ticks will be placed in a hurry, some of them where the test only passes near the requirement. The number grows, and the checking does not.
That is why other questions stand next to coverage in the report:
- What is not covered, and why? A list, not a percentage.
- How important is the uncovered part? The colour of a button, or payment.
- Are the tests on the boundaries? 49.99 and 50.00 €, the 14th and the 15th day.
- What do the defects say? If after the release the bugs come from one place, the tests there are weak, whatever the number says.
The quality of the tests is measured differently: the tests are run against a version of the system with a real bug, and you see whether they fail. Later in the course you will do that with every table and every scenario of yours.
More precisely, if you are curious
The observation that a measure that becomes a target stops being a good measure is known as Goodhart's law (after the economist Charles Goodhart). The wording in this form came later, from the anthropologist Marilyn Strathern.
Exercise
The missing test
1. What to add. The matrices are complete. Which test brings something new?
Test