Skip to content
wannacode

Why programs fail

In short: A person makes an error, the error leaves a defect in the requirement or the code, and a defect that gets executed becomes a failure the customer sees. Testing looks for failures, but it cannot prove that there are no defects.

Problem

Who got it wrong?

The shop launches the promo code "AUTUMN10": 10 € off, once per customer. A week later accounting sees a customer who used it in three orders. 30 € instead of 10 €.

The developer says: "I built it as written". The requirement says "once per order". The analyst says: "That is how the owner told me". The owner is sure they said "per customer".

Who got it wrong? That is the wrong question. It is more useful to see that these are three different things that happen one after another.

The idea

Error, defect, failure

  • An error (mistake) is a human action: someone understands, writes or does something wrong. The analyst writes down "per order".
  • A defect (also bug or fault) is a flaw in a work product, left by the error. The defect here is in the requirement. Then it carries over into the code, because the developer faithfully builds what is written.
  • A failure is the moment the system does something other than expected while it runs. The customer uses the code three times.

The direction is always this: error → defect → failure. The customer sees only the failure. Testing starts from it and works backwards.

Not every defect becomes a failure. If nobody tries the code a second time, the defect sits there unseen.

More precisely, if you are curious

This is how the ISTQB CTFL v4.0.1 syllabus gives them (section 1.2.3, "Errors, Defects, Failures, and Root Causes"). A failure can also come from environmental conditions, with no human error: radiation or an electromagnetic field can damage firmware. In the shop that is rare.

At work the word "bug" means all three, and that is why conversations get muddled. "There is a bug in the promo code" may be the failure the customer saw, or the line of code that causes it.

The idea

The root cause

Why did the analyst get it wrong? Because the client said it on the phone and nobody wrote it down. There was no example either: "Maria orders with the code today and again tomorrow. What happens?" With such an example, "once per customer" and "once per order" would have given different answers out loud.

That is the root cause: why the error happened at all. Fixing the requirement and the code fixes this defect. Fixing the root cause (requirements are written with examples and the owner confirms them) guards against the next ones of the same kind.

The aim of the analysis is not to find someone to blame. The aim is a process that does not allow the error a second time.

Try it

Put the chain in order

Order from the cause to what accounting sees.

  1. The client says on the phone "the code works once per customer", nobody writes it down
  2. A customer uses the code in three orders and gets three discounts
  3. The requirement says "once per order"
  4. The analyst writes from memory "once per order"

Try it

Error, defect or failure?

1. The returns requirement says "within 14 days" without saying from when.
2. The developer is tired and writes "reject over 11 items" instead of "over 10".
3. A customer puts 11 of one product in the cart and places the order.
4. An order that has already shipped can be cancelled, because the code lacks a check of the state. Nobody has tried it yet.

The idea

A test case

How do you catch the failure before the customer does? You put the system in a given state, do something, and compare with what should happen. Written down, that is a test case:

Part For the promo code
Preconditions The customer has already used "AUTUMN10" in one paid order.
Input and actions A new order for 40 €, with the same code.
Expected result The code is refused: "This code has already been used". The total stays 40 €.

The most important part is the expected result, and it comes from the requirement, not from the program. If you copy it from what the program does today, the test will confirm its bugs too.

In the scenarios module the same case will be written as "Given … When … Then …". The idea is the same: a start, an action, an expectation.

More precisely, if you are curious

The ISTQB glossary defines a test case as a set of preconditions, inputs, actions (where applicable), expected results and postconditions, developed based on a test condition. A test condition is what you check ("the promo code works once per customer"), and the test cases are the concrete examples of it. Here a test case is something a person can run. The tests in code from the other courses are the same idea, written for a machine.

The idea

A test shows, it does not prove

The shop has 200 tests for the cart and all of them pass. Does that mean the cart has no defects?

No. It means these 200 cases work. Try listing all of them: 10 of each of 500 products, with a promo code or without, with every total, with every delivery method. There are more combinations than you will run in your life.

This is the first of the seven testing principles: testing can show that defects are present, but not that they are absent. A passing test lowers the chance of a hidden defect. It does not make it zero.

The idea

Six more principles, six stories

2. Exhaustive testing is impossible. Not every combination of the cart can be run. So you choose the most important ones: by risk, by boundaries (49.99 €, 50.00 €), by the techniques of the module "Test design techniques".

3. Early testing saves time and money. The question "and at exactly 50 €?" at the meeting costs a minute. After release it costs a fix, a new test, a new version and angry customers.

4. Defects cluster together. Most defects in the shop are in promo codes and in order cancellation: where the rules are many. That is where you test more.

5. Tests wear out. The same tests, run for months, only catch what they have always caught. The new defects are elsewhere. Tests are updated and new ones added.

6. Testing is context dependent. A shop and hospital software are not tested the same way. Even the website and the mobile app of one shop have different risks.

7. Absence-of-defects fallacy. A system with no known defects can still be wrong: if it does what is written, and what is written is not what is needed. Verification passed, and validation was never done.

More precisely, if you are curious

The principles are from ISTQB CTFL v4.0.1, section 1.3.

In older syllabi the fifth was called the "pesticide paradox": like insects that get used to the poison. Defect clustering (the fourth) is an example of the Pareto principle: a small share of the modules holds a large share of the defects.

Exercise

Which principle?

1. Name the principle. Every story is from the shop. Which of the seven principles does it describe?

1. The shop tests the cart with Chrome on a computer. The mobile app of the same shop is tested with phones, a weak network and a drop in the middle of payment.
2. Seven of last year's ten defects were in promo codes. The tester starts the new cycle there.
3. A set of 40 delivery tests has been green for a year. Last month there were three failures for delivery to an office, which the tests do not cover.
4. The new payment page has no known defect. Customers still give up, because it asks for registration before payment.
5. The question "and at exactly 50 €?" is asked at the requirements meeting. The fix takes a minute instead of a week after release.

Test

Check yourself

1. Which direction is right?
2. All 200 tests pass. What can you say?
3. Which is NOT part of a test case?
4. A defect in code that is never executed is…
5. Why look for the root cause?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·