Skip to content
wannacode

Test levels

In short: Testing is not one thing. At component level one rule is checked on its own, at integration whether the parts talk to each other, at system level the whole behaviour, and at acceptance whether it is what the business wants. Each level catches bugs the others miss.

Problem

Everything passes, and the money does not add up

The shop's developers have 400 tests for delivery and the promo code. They all pass. The tester has walked the whole order path: cart, delivery, payment. Everything is fine.

On the first day after launch, the bank transfers 0.50 € for an order of 50.00 €. The cart sends the total in euros, and the payment service reads it in cents. Each part is right on its own. The bug is between them, and no test looks there.

The idea

Four levels, four questions

Testing is done at several test levels. Each asks a different question and catches different bugs:

Level The question In the shop
Component Is one rule right on its own? The delivery function at 49.99 €, 50.00 € and 50.01 €
Integration Do the parts talk to each other correctly? The cart and the payment service: euros or cents?
System Does the whole do what is described? The whole path: cart, promo code, delivery, payment, email
Acceptance Is this what the business wants? Can it be used? Accounting checks the promo code report

The levels are not better or worse. They are different nets with different holes.

More precisely, if you are curious

ISTQB CTFL v4.0.1 describes the levels in section 2.2.1 and splits them into five: component testing (also called unit testing), component integration testing, system testing, system integration testing (the shop with the courier, with the bank) and acceptance testing. For each level the syllabus gives the test object, the objectives, the test basis (where the test cases come from), typical defects and who is responsible.

The idea

The lower, the more precisely it points

A component test calls one function with 50.00 € and sees 4.90 €. You know exactly where the bug is. It runs in milliseconds and needs no database, browser or bank.

A system test sees the same bug, but through the whole shop. When it fails, you do not yet know whether it is delivery, the promo code or the page. And some cases are hard to reach there: to check 49.99 €, you have to fill a cart for exactly that much.

The opposite is also true. A component test does not see that the cart and the bank count in different units. That is why you need both levels.

More precisely, if you are curious

Component tests are usually written by developers, often before the code (TDD). If you have taken one of the programming courses, you saw them there as tests with JUnit, xUnit, pytest or node:test. Here we write no code: in the automation module you will see the same rules as component tests in the four languages.

Try it

Which level catches it?

1. The function that computes delivery returns 4.90 € for a cart of exactly 50.00 €.
2. The cart sends the total in euros, and the payment service reads it in cents: the customer pays 0.50 € instead of 50.00 €.
3. Everything is computed correctly, but the shop sends the order confirmation only in English, and the customers are in Bulgaria.

The idea

Acceptance is not one thing

Acceptance testing asks "can it be used", but "used" means different things to different people. So there are several kinds:

  • User acceptance testing (UAT): the people who will use the system do their real work with it. The accountant runs the monthly report.
  • Operational acceptance testing: the people who will keep the system running check backup, restore, monitoring and updates.
  • Contractual and regulatory: the yardstick is the contract with the client or the law. Are cookie consents stored? Is everything the contract lists there?
  • Alpha and beta: people outside the team try the product. Alpha is in the developer's environment, beta is at the customers' side, on their phones and networks.

The yardstick for acceptance is the acceptance criteria of the stories. That is why we write them together with the business, before the code.

Try it

What kind of acceptance?

1. The accountant checks whether the monthly promo code report does the job for her.
2. The administrators check that the nightly backup of orders works and that the shop can be restored from it in under an hour.
3. 200 real customers use the new mobile app from home for two weeks before the official launch.
4. It is checked that the shop stores the customers' cookie consents, as the data protection law requires.

The idea

The level is the goal, not the time

It is easy to picture the levels as stairs: first component, then integration, acceptance at the end. In the waterfall model it often is like that. But in an Agile team all four can happen for one story in one week: the developer writes a component test, QA runs the scenario on the system, and the product owner accepts it at the demo.

You tell a level by its goal: what you check and against what. Not by when, or by who.

More precisely, if you are curious

Levels go together with test types (the next lesson). There is functional and non-functional testing at every level: the performance of one function is at component level, and the performance of the whole shop with 200 customers is at system level.

Exercise

Levels in the shop

1. Which level catches the bug? For each bug, choose the lowest level at which it shows.

1. The promo code AUTUMN10 is still accepted on the day after it expires.
2. The courier changed the address format. Orders still reach them, but with an empty city field.
3. Every step works on its own, but if the customer goes back from payment to the cart, the promo code disappears.
4. The warehouse says that with the new returns screen, handling one parcel takes 6 steps instead of 2, and they cannot keep up.

Test

Check yourself

1. Which level answers the question "is this what the business wants"?
2. Why is system testing alone not enough?
3. What does integration testing check?
4. Who usually does user acceptance testing?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·