Testing from experience
In short: Experience knows where bugs usually are: zero, negative, very large, double click, midnight, leftovers of old logic. Error guessing turns that into attacks. Exploratory testing learns the system while testing it, by a charter, in a time-boxed session, with notes. A checklist keeps what the team has learned. All three find what the rule does not say, which is why they complement the other techniques.
Problem
Ten minutes, three bugs
Delivery has been tested by partitions and boundaries. Discounts by a decision table. The order by its transitions. Everything passes.
A colleague from support, who has worked at three online shops, sits down with it for ten minutes. She presses "Pay" twice quickly: two orders, two payments. She enters the code with a space in front: "The code has expired", though it has not. She puts 150 € in the cart and pays 4.90 € for delivery.
None of the three bugs comes from a rule. They come from experience of what usually breaks.
The idea
Experience-based techniques
The last group of techniques does not start from the rule and does not start from the code. It starts from the tester's experience: what bugs they have seen, where, in what systems.
- Error guessing: tests aimed at the places where, from experience, there are usually bugs.
- Exploratory testing: you learn the system while testing it, and each result steers the next test.
- Checklist-based testing: the team's experience, written down as questions.
They find what the rule does not say. Their weakness is that they depend on the person and give no measure of "how much is covered". That is why they complement the techniques from the previous lessons rather than replace them.
The idea
Error guessing
Experienced testers carry a list in their head: where there is usually a bug. A list for a shop:
- zero and empty: an empty cart, an empty code field, 0 items;
- negative and absurd: -1 item, a total of 999 999 €;
- very large: an order of 250 €, when all tests are up to 80 €;
- text written differently: a code in lowercase, with a space in front, pasted with a newline;
- a repeated action: a double click on "Pay", "back" after paying, the same code in two tabs;
- time: midnight on the code's last day, time zones, 29 February;
- leftovers: old logic nobody removed ("from 100 € 4.90 again");
- what stays after an error: does a refusal empty the cart, is half an order left behind.
When the list is run systematically against the system, every line becomes an attack (fault attack). Some of them open questions the rule does not answer: should a lowercase code be valid? That is not the tester's decision, it is the owner's.
More precisely, if you are curious
ISTQB CTFL v4.0.1 describes the experience-based techniques in section 4.4: error guessing (with fault attacks as a systematic way to apply it), exploratory testing and checklist-based testing. The list of guesses comes from the tester's experience, from the defect history of the system and from known kinds of bugs in similar systems.
Try it
Attack: zero and a lot
From the list of attacks take "zero" and "very large". Write them as delivery tests.
The rule: Delivery is free for an order from 50 € inclusive, otherwise it is 4.90 €. An empty cart cannot be ordered.
| Input | Expected | The correct system | |||
|---|---|---|---|---|---|
| # | Cart total | Refused | Delivery | ||
An empty result cell is not checked. Amounts are written like 49.99.
Solution
| Cart total | Refused | Delivery |
|---|---|---|
| 0 | yes | |
| 250 | no | 0 |
0 € catches the empty cart that pays delivery. 250 € catches the leftover of the old tariff, where from 100 € delivery is 4.90 € again. By the rule 70 € and 250 € are one partition, so partitions and boundaries do not suggest the second test. Experience does: "everything over something" often has another "over something" forgotten in the code.
The idea
Exploratory testing
In exploratory testing you do not write the test cases in advance. You learn the system, design and run the tests at the same time. If you see something odd, the next test goes there.
It does not mean "clicking at random". Three things make it work rather than a stroll:
- A charter: one goal for the session. A handy template: "Explore [what] with [what] to discover [what]." For example: "Explore the promo code with expired, used and lowercase codes to discover how the shop tells the customer why the code does not apply."
- A time-boxed session: usually 60 to 90 minutes, without interruption, only on the charter. If you see something outside it, you note it for a later session.
- Notes: what you tested, how, what you found, what questions you have. At the end there is a debrief with the test lead: what was found, what remains, what the next charter is.
More precisely, if you are curious
The template "Explore … with … to discover …" is from Elisabeth Hendrickson's book "Explore It!" (2013). Testing in sessions with charters and debriefs (session-based test management) was described by Jonathan and James Bach around 2000. CTFL v4.0.1 says exploratory testing is most useful when the specification is poor or inadequate, time is short, or as a complement to more formal techniques. And that it works best with experienced testers who know the domain.
Try it
The charter and the notes
The idea
Checklist-based testing
A list of attacks in one colleague's head helps one team. Written down, it helps everyone. Checklist-based testing runs a list of questions or conditions against the system.
A checklist for every new form in the shop:
- an empty field, the longest allowed text, one character more;
- spaces at the start and end, lowercase and uppercase;
- Cyrillic, Latin, special characters;
- a double click on the button, "back" after submitting;
- with the keyboard, without a mouse; with a screen reader;
- the error message says what to fix.
It is a relative of the review checklist, only for the running system rather than the document. And it has the same risk: a stale checklist is followed mechanically, and new bugs slip past it. That is why it is updated with every defect that got away.
Try it
The session
Order an exploratory session from the charter to the next step.
The charter: what, with what, and what you are looking forThe debrief: with the test lead, what was found and what remainsThe next step: defect reports, questions for the owner, new chartersThe session: 60 to 90 minutes without interruption, only on the charterThe notes: what was tested, findings, questions, ideas
The charter comes before the session, the notes during it, the debrief after it. The findings from the debrief become defect reports, questions for the owner and new charters. That way exploring does not end with "I had a look around".
Exercise
Attacks on the limit
1. Negative, and what stays. The two attacks from the list: a negative quantity, and "what stays in the cart after a refusal".
Write the attacks on the cart limit: a negative quantity and adding over the limit. Check what stays in the cart too.
The rule: The cart can hold at most 10 of one product. Adding beyond 10 is refused with a message, and the cart stays the same. You add at least 1 item.
| Input | Expected | The correct system | ||||
|---|---|---|---|---|---|---|
| # | In the cart | Adding | Refused | In the cart after | ||
An empty result cell is not checked. Amounts are written like 49.99.
Solution
| In the cart | Adding | Refused | In the cart after |
|---|---|---|---|
| 3 | -1 | yes | 3 |
| 8 | 5 | yes | 8 |
(3, add -1) catches the version that refuses only 0 and lets a negative quantity remove items. (8, add 5) is refused, and the cart has to stay at 8. If you do not check what stays, the version that empties the cart on a refusal passes: the refusal is there, and the customer loses everything they picked.
Test