Testing in the life cycle
In short: The development model changes when you test, not whether. In waterfall testing is a phase at the end, in Agile it is in every story, in DevOps it is in the pipeline on every change. Shift-left moves checking towards the requirement. The pyramid says how many tests at which level, and the quadrants which kinds.
Problem
The testers got the shop in the last month
The project for the new shop is on plan: 3 months of requirements, 6 months of programming, then testing. In the last month the testers find 140 defects. 30 of them are not in the code but in the requirements: "within 14 days" without saying from when, "over 50 €" without saying whether it is inclusive.
Fixing a requirement in month 10 means touching code written in month 4. The release slips by 3 months.
The idea
The model changes when, not whether
Every software development life cycle model has testing. What differs is when and how:
| Model | Where testing is |
|---|---|
| Waterfall | A separate phase at the end, after all the code. |
| V-model | Each phase has its own test level, planned from the start: requirements → acceptance, architecture → integration. |
| Iterative, incremental | In each iteration the new part is tested, and everything old with regression tests. |
| Agile (Scrum, Kanban) | In every story. QA is in the team, and "done" means tested too (Definition of Done). |
| DevOps | In the pipeline (CI/CD): every change goes automatically through the tests before it reaches customers. |
One rule holds for all of them: for every development activity there is a matching test activity. A requirement has a review, code has a test, a story has acceptance criteria.
More precisely, if you are curious
This is CTFL v4.0.1, section 2.1. The syllabus lists good practices that hold for every model: every development activity has a corresponding test activity; every test level has its own objectives; test analysis and design start during the corresponding development phase; testers review work products as soon as drafts are available. DevOps is in 2.1.4, and shift-left in 2.1.5.
Try it
Which model?
The idea
Shift-left: to the left in time
Draw the project as a line: requirement, design, code, test, release. Shift-left means moving checking to the left, towards the start.
In the shop these are things you have already seen in the course:
- reviewing the requirement "free delivery over 50 €" before the code: is 50.00 € included?
- Example Mapping and examples for every rule, before the story goes into work;
- scenarios written before the code, which later become automated tests;
- component tests the developer runs on every change.
Shift-left does not mean less testing at the end. It means that at the end you find less, because most defects were caught where they cost least.
More precisely, if you are curious
Shift-left has a price: training, time at the start, automation. CTFL v4.0.1 says so too: the approach needs effort early on, and the savings come later. The opposite idea, shift-right, is testing and monitoring after release: measuring with real customers, rolling out new features gradually. That is what the "Validating with the client" module is about.
The idea
The pyramid: how many tests at which level
The test pyramid says how many tests to have at each level:
- The base: many component tests. Thousands, run in seconds, pointing exactly at the bug.
- The middle: fewer integration and service tests. They check the contracts between the parts.
- The top: a few end-to-end tests through the whole shop and the browser. They are slow, break when one button changes, and when they fail you do not yet know where the bug is.
The higher you go, the slower, more expensive and more fragile the tests, and the fewer of them there should be. The upside-down pyramid, with thousands of browser tests and a few component tests, is called the ice cream cone.
More precisely, if you are curious
The pyramid is Mike Cohn's ("Succeeding with Agile", 2009). It is not a law: the number of levels and their names vary. CTFL v4.0.1 describes it in section 5.1.6 as a model showing that tests at lower levels are smaller, faster and more numerous, and at higher levels slower, bigger and fewer.
The idea
The quadrants: which kinds
The pyramid counts. The testing quadrants arrange the types by two questions: is the test business-facing or technology-facing, and does it support the team while building or critique the finished product?
| Support the team | Critique the product | |
|---|---|---|
| Business | Q2: scenarios and examples, prototypes, functional tests | Q3: exploratory testing, usability, UAT, demos |
| Technology | Q1: component and integration tests | Q4: performance, load, security |
Q1 and Q2 are usually automated. Q3 is manual: a person looks at the product. Q4 is done with tools. A team that only has Q1 has fast code but does not know whether it is the right product.
More precisely, if you are curious
The quadrants are Brian Marick's (2003), made popular by Lisa Crispin and Janet Gregory's books on Agile testing. In CTFL v4.0.1 they are in section 5.1.7.
Exercise
The quadrants in the shop
1. Which quadrant? Sort the shop's tests.
Test