The traceability matrix
In short: Traceability is the link between requirement, test and defect, in both directions. The matrix is a table: the requirements are rows, the tests are columns, and a tick says "this test checks this requirement". An empty row is an uncovered requirement, an empty column is a test with no requirement, and two tests with the same ticks often check the same thing.
Problem
Who tests the code's expiry?
Two weeks before the autumn campaign, marketing asks: "The code AUTUMN10 expires on 31 October. Has someone checked that it does not work on 1 November?"
QA has 40 tests for the promo codes. Which one checks the expiry? Someone opens the folder and starts reading. An hour later the answer is "probably none". Five tests check the 10 %, three check the discounted products, and there is not a single one for the expiry.
The tests were there, but nobody could say what they cover.
The idea
Traceability
Every requirement should have at least one test that checks it. Every test should have a reason, that is, a requirement. And every defect is a failure of some test against some requirement.
The link between these three is called traceability. It is followed in both directions:
- Forward: from the requirement to the tests. It asks "has this been checked?". That way marketing would have had an answer in a minute.
- Backward: from the test or the defect to the requirement. It asks "why does this test exist?" and "which rule does this defect break, and how important is it?".
The requirement "A code has an expiry date" has a number, R-4. The test has a number, T-12. The defect has a number, D-31. Traceability is the record of which number is linked to which.
More precisely, if you are curious
ISTQB CTFL v4.0.1 (section 1.4.4) calls this traceability between the test basis (the requirements, stories, criteria) and the testware (the tests, results, defects). According to the syllabus, good traceability makes it possible to evaluate coverage, supports impact analysis when things change, makes reports easier to understand and makes audits easier.
In the IIBA BABOK v3, tracing is a task in the Requirements Life Cycle Management knowledge area: Trace Requirements. There the links are also between the requirements themselves: which was derived from which (derive), which depends on which (depends), which solution satisfies which requirement (satisfy), and which test confirms it (validate). Here we look mainly at the last one.
IREB CPRE also covers traceability in its chapter on requirements management: backward to the source (who asked for it and why) and forward to what follows from it (design, code, tests).
The idea
The matrix
The simplest record is a table. The requirements are rows, the tests are columns, and a tick means "this test checks this requirement". This is the traceability matrix.
| T-1 | T-2 | T-3 | T-4 | |
|---|---|---|---|---|
| R-1 10 % off | ✓ | ✓ | ||
| R-2 not for discounted products | ✓ | |||
| R-3 one paid order | ✓ | |||
| R-4 expiry |
- T-1: a T-shirt for 20.00 € with AUTUMN10, 18.00 €.
- T-2: a T-shirt for 20.00 € and discounted jeans for 40.00 € with AUTUMN10, the discount is 2.00 €.
- T-3: a second try with the same code, "The code has already been used".
- T-4: a loyal customer, a T-shirt for 20.00 €, 19.00 €.
The table reads in seconds. T-2 checks two requirements: that is normal. The row of R-4 is empty. So is the column of T-4. These are the two holes the matrix shows straight away.
Defects are linked to the same numbers: "D-31: an expired code is accepted, R-4". At work this table lives in the test management tool or in an ordinary spreadsheet. The idea is the same.
The idea
What the matrix shows
Three things are visible at a glance:
- An uncovered requirement: an empty row. Nothing checks the rule. If you break it, no test fails.
- A test with no requirement: an empty column. Either the requirement is missing from the list (the shop has loyalty, but this list does not), or the test checks something nobody asked for. Either way, someone has to decide.
- Duplicate coverage: two tests with the same ticks that check the same thing. "A cart of 35.00 €" and "A cart of 20.00 €" follow the same rule and give the same answer. One of them brings nothing new, yet it is maintained like the other.
Be careful with duplicates: identical ticks are only a hint. "A cart of 49.99 €" is in the same row as "A cart of 20.00 €", but it is not a duplicate. 49.99 € is just below the boundary, and that is where the bugs live.
The share of requirements with at least one test is the requirements coverage. Four requirements, three with a test: 75 %. In the third lesson of the module you will see why 100 % does not mean "well checked".
Try it
Read the matrix
Try it
The promo code matrix
Mark which test checks which requirement. Then see what has been left without a link.
Tests
- T-1: A T-shirt for 20.00 € with AUTUMN10: 18.00 €.
- T-2: A T-shirt for 20.00 € and discounted jeans for 40.00 € with AUTUMN10: the discount is 2.00 €.
- T-3: Maria has paid an order with AUTUMN10. In a new order she enters AUTUMN10: "The code has already been used".
- T-4: A loyal customer, a T-shirt for 20.00 €: 19.00 €.
- T-5: Maria has entered AUTUMN10 but has not paid. In a new order the code works.
Press the cell where the test checks the requirement; press again to remove the link. The arrow keys move between the cells, Space or Enter marks.
| Requirements | T-1: A T-shirt for 20.00 € with AUTUMN10: 18.00 €. | T-2: A T-shirt for 20.00 € and discounted jeans for 40.00 € with AUTUMN10: the discount is 2.00 €. | T-3: Maria has paid an order with AUTUMN10. In a new order she enters AUTUMN10: "The code has already been used". | T-4: A loyal customer, a T-shirt for 20.00 €: 19.00 €. | T-5: Maria has entered AUTUMN10 but has not paid. In a new order the code works. | Covered by |
|---|---|---|---|---|---|---|
| R-1 A promo code gives 10 % off. | 0 | |||||
| R-2 A promo code does not apply to discounted products. | 0 | |||||
| R-3 A code can be used in one paid order. A second try: "The code has already been used". | 0 | |||||
| R-4 A code has an expiry date. After it: "The code has expired". | 0 | |||||
| Checks | 0 | 0 | 0 | 0 | 0 |
The reference matrix
- R-1: T-1, T-2
- R-2: T-2
- R-3: T-3, T-5
- R-4: uncovered
- T-4: no requirement
T-2 checks two requirements: the 10 % on the T-shirt and that the discounted jeans get no code discount. The expiry (R-4) has no test: that is the hole marketing is asking about. And T-4 is about loyalty, which is not in this list. Someone has to decide: is a requirement added, or is the test in the wrong folder?
The idea
Who maintains the matrix
The matrix is worth something only if it is right. That is why it is written together with the tests, not at the end:
- A requirement gets a number when it enters the list.
- A test gets a number and a tick for its requirement when it is written. A test with no tick is a signal, not finished work.
- A defect gets the number of the test and of the requirement when it is recorded.
The numbers have to be stable. If R-4 becomes R-7 tomorrow, all the links get mixed up. That is why a number is never reused, even when its requirement is removed.
The analyst is responsible for the requirements and their numbers, QA for the tests and their links. But the matrix is shared: the owner can also see from it what has been checked.
Exercise
The delivery matrix
1. Delivery and the cart. Five requirements, seven tests.
Mark the links. Then find the two tests that check the same thing and mark one of them as redundant.
Tests
- T-1: A cart of 49.99 €: delivery is 4.90 €.
- T-2: A cart of 50.00 €: delivery is free.
- T-3: The cart holds 9 T-shirts. Adding 2 more is refused, 9 stay.
- T-4: A cart of 35.00 €: delivery is 4.90 €.
- T-5: A cart with no products: the order is refused.
- T-6: A cart of 20.00 €: delivery is 4.90 €.
- T-7: Paying by card: the order becomes paid.
Press the cell where the test checks the requirement; press again to remove the link. The arrow keys move between the cells, Space or Enter marks.
| Requirements | T-1: A cart of 49.99 €: delivery is 4.90 €. | T-2: A cart of 50.00 €: delivery is free. | T-3: The cart holds 9 T-shirts. Adding 2 more is refused, 9 stay. | T-4: A cart of 35.00 €: delivery is 4.90 €. | T-5: A cart with no products: the order is refused. | T-6: A cart of 20.00 €: delivery is 4.90 €. | T-7: Paying by card: the order becomes paid. | Covered by |
|---|---|---|---|---|---|---|---|---|
| R-1 Delivery is free for an order from 50.00 € inclusive. | 0 | |||||||
| R-2 Under 50.00 € delivery is 4.90 €. | 0 | |||||||
| R-3 An empty cart cannot be ordered. | 0 | |||||||
| R-4 The cart holds at most 10 of a product. | 0 | |||||||
| R-5 The customer sees the delivery cost in the cart before paying. | 0 | |||||||
| Checks | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
The reference matrix
- R-1: T-2
- R-2: T-1, T-4, T-6
- R-3: T-5
- R-4: T-3
- R-5: uncovered
- T-7: no requirement
Redundant: T-4 / T-6 (35 € and 20 € are in the same group: under 50 €, far from the boundary. “T-1” stays, because 49.99 € is right on the boundary.)
R-5 (the delivery cost before the customer pays) has no test, and T-7 (payment) has no requirement. T-4 and T-6 are in the same group: under 50 €, far from the boundary. T-1 is in the same row, but it is not a duplicate, because 49.99 € is right on the boundary. If the threshold had become "over 50 €" instead of "from 50 €", T-2 would catch it. If it had become "from 49 €", T-1 would catch it. T-4 and T-6 would catch neither.
Test