Pairwise testing
In short: All combinations grow as a product: 3 × 3 × 3 × 3 = 81. Pairwise testing only asks that every pair of values of two parameters meets in at least one test, and here 9 are enough. Experience shows most bugs depend on one or two parameters. Those that depend on three at once, pairs may miss.
Problem
27 tests or 9
The shop starts selling in Greece and Romania. The tester has to check the order with every payment method, every delivery method and every country:
- payment: card, cash on delivery, bank transfer;
- delivery: to the address, to a courier office, to a parcel locker;
- country: Bulgaria, Greece, Romania.
All combinations are 3 × 3 × 3 = 27. One manual order from start to finish takes 10 minutes. Four and a half hours. Tomorrow a fourth parameter arrives, the device (phone, tablet, computer), and it becomes 81.
Exhaustive testing is impossible here too, only faster.
The idea
Pairs
Most bugs do not depend on everything at once. They depend on one value ("bank transfer does not work") or on two ("cash on delivery does not work for Romania"). Bugs that show only with three exact values at once are rarer.
Pairwise testing builds on that. It does not ask for all combinations. It asks that every pair of values of every two parameters meets in at least one test. Card with office, card with Romania, office with Romania, and so on for all pairs.
Here 9 tests are enough, for three parameters and for four:
| # | Payment | Delivery | Country | Device |
|---|---|---|---|---|
| 1 | card | to address | Bulgaria | phone |
| 2 | card | to office | Greece | tablet |
| 3 | card | to locker | Romania | computer |
| 4 | cash on delivery | to address | Greece | computer |
| 5 | cash on delivery | to office | Romania | phone |
| 6 | cash on delivery | to locker | Bulgaria | tablet |
| 7 | bank transfer | to address | Romania | tablet |
| 8 | bank transfer | to office | Bulgaria | computer |
| 9 | bank transfer | to locker | Greece | phone |
Take any two columns. Each of their nine pairs is somewhere in the table, exactly once. 81 combinations, 9 tests.
Try it
How many?
The idea
What they miss
Pairs are a bet. If the bug shows only with three values at once (cash on delivery, to locker, Romania), and that triple is not among the nine rows, it slips through.
So the technique is not for everything:
- for many, nearly independent parameters it is excellent: settings, browsers and devices, payment and delivery methods;
- for a rule with a number there are boundary values, for combinations the rule lists there is the decision table, for a path there is the state machine;
- for risky triples tests are added by hand, or a table of triples is built.
There are also impossible combinations. If, say, cash on delivery to a locker is not offered, that pair is forbidden, and the table is built without it. And a separate test checks that the shop really does not allow it.
The table above is easy because all parameters have three values each. In real life they do not, and tables are built with a tool, not by hand.
More precisely, if you are curious
In ISTQB CTFL v4.0 pairwise testing is not part of the syllabus: there the black-box techniques are four (partitions, boundaries, decision tables, transitions). Combinatorial techniques, such as pairwise testing and the classification tree, are in the higher-level syllabi, for the test analyst (Advanced Level Test Analyst). They are here because they are used every day, and because they answer the question "what do I do when the decision table becomes 1024 columns".
The idea that most failures are triggered by the interaction of few parameters is supported by NIST studies of real systems. There are many tools for pairwise tables: for example Microsoft's PICT, which is open source. They also support constraints (forbidden pairs) and higher strengths: every triple, every quadruple.
The table in the lesson is an L9 orthogonal array: 9 rows for 4 parameters with 3 values each. In it every pair appears exactly once. Tools usually produce tables where some pairs repeat, because the parameters have different numbers of values.
Exercise
The missing pair
1. The broken table. Someone changed one row of the pairwise table.
Test