The plan and risk-based testing
In short: There is always less time for testing than needed. The plan says what, how, who and when; entry and exit criteria say when to start and when to stop. Risk, likelihood times impact, orders what to test first and how deeply.
Problem
Four days for two weeks of work
The new version of the shop has a new promo code, a changed delivery threshold and fixes in payment. The testers say: we need two weeks for everything. The release is in four days, and it will not move, because the advertising for the sale is already running.
What will you test? If you start from the top of the list, on day four you will still be checking the "About us" page, and the promo code will go out untested.
The idea
The plan: what, how, who, when
The test plan is the team's agreement about testing. It is not a form but answers to questions:
- What is tested and what is not (scope), and why;
- how: the levels, types, techniques, what is automated;
- who and with what: the people, the environment, the data, the tools;
- when: the schedule and the entry and exit criteria;
- the risks and what is done about them;
- how it is reported: who sees what, and when.
In a waterfall project the plan is a document. In an Agile team it is half a page and a conversation at the start of the iteration. The questions are the same.
More precisely, if you are curious
The test plan is CTFL v4.0.1, section 5.1.1. A test plan document following ISO/IEC/IEEE 29119-3 has more fields (context, communication, risk register, strategy, staffing, schedule). The exam asks about the purpose of the plan and its typical content, not about an exact template.
The idea
How long will it take?
Estimating testing is not guessing. There are a few simple techniques:
- Ratio-based: in past versions testing was 30% of programming time. Programming is 20 days now, so testing is about 6.
- Three-point: you ask for the best case (a), the most likely (m) and the worst (b). The estimate is (a + 4m + b) / 6. The worst case pulls it up, and that is honest.
- With the team: everyone estimates alone, then you compare and discuss the difference. Planning Poker cards in Agile work this way.
An estimate is an estimate. You state it with its assumption: "5 days, if the environment is ready on Monday".
More precisely, if you are curious
CTFL v4.0.1, section 5.1.4, lists four techniques: estimation based on ratios, extrapolation, Wideband Delphi (of which Planning Poker is a variant) and three-point estimation. For three points the syllabus also gives the standard deviation: (b − a) / 6.
Try it
Estimate
The idea
When you start and when you stop
Testing has no natural end: there are always more defects. So entry and exit criteria are agreed:
- Entry: when it makes sense to start. The version is in the test environment, the component tests pass, the test data is loaded, the stories have acceptance criteria.
- Exit: when you stop. All high-risk tests pass; there is no open defect of critical severity; the regression suite is green; the remaining risks are written down and accepted.
A criterion is checked, not felt. "We are ready" is an opinion, "0 open critical defects" is a fact.
In an Agile team the exit criteria of a story are in the Definition of Done, and the entry criteria in the Definition of Ready.
Try it
Can we start? Can we stop?
The idea
Risk says what comes first
Exhaustive testing is impossible, and there is not enough time. The question is what to test first and most deeply. The answer is risk.
A risk is something that may happen and will do harm. It has two measures:
- likelihood: how likely is a defect here? New code, complex code, code changed in a hurry, parts with many defects so far;
- impact: how much harm if there is one? Money, many customers, the law, reputation.
The risk level is likelihood times impact. The promo code, rewritten from scratch (likelihood 5), which can give money away (impact 4), is 20. The "About us" page with a new phone number is 1.
Risk-based testing orders the tests by this number: high risk is tested first, with more techniques and more cases; low risk last, or only with a quick look. If time runs out, what is left untested is the least dangerous.
More precisely, if you are curious
Risk is CTFL v4.0.1, section 5.2. The syllabus distinguishes product risks (the product not doing what it should: a wrong total, a slow cart) from project risks (something getting in the way of the work: a late environment, a person off sick). Product risk analysis has two parts, identification and assessment, and the risks are controlled by mitigation (testing) and monitoring. The 1–5 scales here are an example: in practice there are also scales like low, medium, high.
Exercise
Risk-based testing
1. Order by risk. The four days from the start. Order the parts of the shop in the order you will test them: the highest risk (likelihood times impact) at the top.
Order from the highest risk to the lowest.
Delivery: the threshold was changed (likelihood 4, impact 3)Card payment: untouched, but everything goes through it (likelihood 2, impact 5)Cart limit: untouched (likelihood 2, impact 2)Promo code: rewritten in this version (likelihood 5, impact 4)"About us" page: the phone number was changed (likelihood 1, impact 1)Returns: untouched (likelihood 1, impact 3)
The promo code is 5 · 4 = 20, delivery 4 · 3 = 12, payment 2 · 5 = 10, the limit 2 · 2 = 4, returns 1 · 3 = 3, "About us" 1 · 1 = 1. Payment is third even though it is the most harmful: nobody touched it. But it does not drop out of the plan: its regression tests are automated and run on every change.
Test