Test types
In short: The level says where you test, the type says what and how. Functional testing checks what the system does, non-functional how well. Black box looks at behaviour, white box at structure. After a fix you run a confirmation test for the defect and regression tests for everything around it.
Problem
They fixed one thing and broke another
A customer complains: an order for exactly 50.00 € pays for delivery. The developer fixes the comparison, the tester tries 50.00 € and sees 0.00 €. Fixed.
The next day accounting sees that the promo code is now deducted twice. The fix changed the function for the total, which the promo code also uses. Nobody ran the promo code tests, because "we touched delivery".
The idea
The type is what and how
The previous lesson said where you test: component, integration, system, acceptance. The test type says what you check and where the test cases come from:
- Functional testing: what the system does. The fee at 50.00 €, refusing an empty cart.
- Non-functional testing: how well. Speed, accessibility, security, reliability: the eight characteristics of ISO/IEC 25010 besides functional suitability.
- Black box: the test cases come from the specification and the behaviour. The code is not looked at.
- White box: the test cases come from the structure: the branches of the function, the paths through the process.
Types combine with levels. A functional black-box test at system level, a non-functional test at component level: any pair is possible.
More precisely, if you are curious
In ISTQB CTFL v4.0.1 the types are in section 2.2.2: functional, non-functional, black-box and white-box. The syllabus says explicitly that every type can be done at every level, with a different focus. The black-box and white-box techniques (equivalence partitions, boundaries, branch coverage) are in the test design module.
Try it
What kind of test?
The idea
After the fix: two questions
When a defect is fixed, you run tests for two different reasons:
- Confirmation testing: is the defect fixed? You run the test that showed it, with exactly the same steps. 50.00 € → 0.00 €.
- Regression testing: did the fix break something that used to work? You run the tests for everything the change can reach. The total is used by the promo code, loyalty and payment. So their tests too.
A bug in something that used to work is called a regression. It is the nastiest kind, because nobody looks there: "we touched delivery, not the promo code".
More precisely, if you are curious
Confirmation and regression testing are in CTFL section 2.2.3. The regression suite grows with every version, so the syllabus recommends automating it and selecting from it by impact analysis: which parts depend on the changed one. The traceability matrix from the Traceability module answers exactly that question.
Try it
What do you run after the fix?
The idea
The system that already runs
The shop is live. From now on every change is in a running system with real customers and data. Testing in it is called maintenance testing. There are three triggers:
- modification: a fix, a new rule (the threshold becomes 60 €), an updated library, a new operating system version;
- migration: a new platform, a new server, moving data from the old shop;
- retirement: the old app is shut down, and the data must stay readable for years.
How widely you test depends on two questions: how big the risk of the change is, and how big the system around it is. That is why maintenance always starts with an impact analysis: what does the change touch?
More precisely, if you are curious
Maintenance testing is CTFL section 2.3. For migration the syllabus also mentions checking the converted data (data conversion), and for retirement testing the archive and the restore procedures, when data has to be kept for a long time.
Exercise
Types and triggers
1. Is it maintenance testing? For each change in the running shop, say whether it is a reason to test, and why.
Test