Why early
In short: A defect in a requirement costs one word if it is caught while reading, and a week of work if it is caught after release. Static testing (reviews and static analysis) looks for defects without running the program. It finds the defect itself, while dynamic testing sees the failure. The two do not replace each other, they complement each other.
Problem
One word, five times
Remember the requirement "free delivery over 50 €", when the owner wanted "from 50 € inclusive". The defect is one word. Let us follow how much it costs depending on when you find it.
- While reading the requirement: the analyst asks "and at exactly 50.00 €?". The fix is one word. Five minutes.
- In the conversation about the story: the example "50.00 € → free" is added. Five more minutes.
- While testing the finished program: the developer fixes the code, the tester tests again, the release slips by a day.
- After release: customers have paid 4.90 € they did not owe. Money is refunded, an email goes out, the fix is urgent, and some of them no longer trust the shop.
The defect is the same. The work built on top of it grows with every step.
The idea
The cost of a defect by phase
The later a defect is found, the more things have to be fixed along with it. The requirement has already become code, the code tests, the tests a report, and the program orders. This is called the cost of a defect by phase.
| When it is found | What has to be done again |
|---|---|
| while reading the requirement | the word in the document |
| in the conversation about the criteria | the word and one example |
| while testing the finished program | the code, the tests, a new release |
| after release | all of the above, plus the money, the customers and the trust |
That is why testing starts early. Not when there is a program, but when there is the first sentence. It is also one of the seven principles: early testing saves time and money.
More precisely, if you are curious
ISTQB CTFL v4.0.1 says in the chapter on static testing that defects found early are often much cheaper to fix than those found late in the life cycle. The same idea is behind the word shift-left, which you will meet later in the course.
Exact multipliers are quoted often: "10 times in the code, 100 times after release". Be careful with them. They come from studies decades old (the best known is linked to Barry Boehm), and their numbers are disputed. For example, Laurent Bossavit in "The Leprechauns of Software Engineering" traces where they come from and how little data stands behind them. The direction is every team's experience. The exact multiplier depends on the project.
The idea
Static and dynamic
There are two ways to look for defects.
- Dynamic testing: you run the program and watch what it does. The tester puts 50.00 € in the cart and sees 4.90 €. That is a failure. Then someone has to find the defect that causes it.
- Static testing: you check a work product without running it. You read the requirement, the model, the code or the tests, and you find the defect itself, directly.
Static testing has two forms. A review: people read and comment. Static analysis: a tool reads the code or the model and looks for known problems, for example a variable that is never used, or code the program can never reach.
Some things are only caught statically: a defect in a requirement before there is code; two requirements that contradict each other; code that works but nobody can maintain. Others are only caught dynamically: a slow page with 500 customers, or two systems that do not understand each other. That is why the two complement each other.
More precisely, if you are curious
CTFL v4.0.1 (chapter 3) lists the differences: static testing finds defects directly, while dynamic testing causes failures from which the defect is then traced. Static testing can also be applied to work products that cannot be run (documents, models), and can catch qualities that do not depend on execution, such as maintainability. Dynamic testing can catch things that only show while running, such as performance.
Static analysis usually runs automatically on every change of the code. The compiler that says "this variable is not initialised" is already such a tool.
Try it
Static or dynamic?
The idea
What gets reviewed
Almost anything that can be read and understood:
- requirements, stories and acceptance criteria;
- models: processes, state machines, decision tables;
- code;
- test cases and scenarios;
- plans, contracts, user manuals.
Reviewing a requirement is special. It is both verification (is it written correctly, unambiguous, verifiable) and validation (is this the need). When the owner reads "over 50 €" and says "no, I wanted from 50 €", that is validation. When the tester asks "and at exactly 50.00 €?", that is verification.
A review also has a limit. It compares the work product with what people know. If nobody can say what the system has to do, the review has nothing to compare against. Then the first job is the conversation, not the reading.
Try it
Review the criteria
The criteria for cancelling an order, before work starts. Mark the problems and choose the kind of each.
Press the words that are a problem; press again to unmark. The arrow keys move from word to word.
The problems in the text
- “at any time” contradicts: C-2 says a shipped order cannot be cancelled. Both cannot be true for the same shipped order. The developer will build one, the tester will check the other.
- “cannot be cancelled” contradicts: It contradicts C-1. The shop's rule is "a paid order can be cancelled before it is shipped", so C-1 is the one to fix: "until it is shipped".
- “quickly” unverifiable: In how many days? Without a number no test can fail. For example: "within 14 days of the cancellation".
Two minutes of reading saved the bug from the lesson "Acceptance criteria": cancelling an order that is already with the courier. Fixed: "C-1. The customer can cancel their order as long as it has not been shipped." C-2 stays. "C-3. When a paid order is cancelled, the money is refunded within 14 days."
The idea
Early feedback
There is an even simpler way to catch the "over 50 €" defect early: show something to the owner. An example, a prototype of the cart, a table of delivery fees for 49.99, 50.00 and 50.01 €. They look at it and say "this one is wrong".
Frequent feedback from stakeholders protects against the most expensive defect: a product built right, but not the one that was needed. The less often you ask, the more work goes in the wrong direction before anyone notices.
That is why reviews are not only for QA. The owner, the warehouse and accounting read the requirements that concern them. They will see things you cannot.
More precisely, if you are curious
CTFL v4.0.1 calls this early and frequent feedback from stakeholders (section 3.2). It helps requirements be understood the same way and problems be seen early. Without it the team risks building a product that does not meet the need, and finding out only at acceptance.
Exercise
What it costs
1. The path of the defect. Order the moments when the "over 50 €" defect can be caught, from the cheapest to the most expensive.
Order from the cheapest to the most expensive place where the defect can be caught.
While testing the finished program: a cart of 50.00 €In the conversation about the criteria: example "50.00 € → free"After release: a complaint from a customer who paid 4.90 €While reading the requirement: "and at exactly 50.00 €?"In the code review: "> 50" is not "from 50 inclusive"
Each line carries everything from the one before it. In the first, the defect is only in the text. In the last, it is in the code, in the tests, in the orders and in the customers' emails.
Test