Defects and the defect report

In short: A defect report is a letter to someone who did not see the bug. They must be able to repeat it the first time: short steps, the expected result from the rule and the actual result from the system. Severity says how much harm the defect does, priority how soon to fix it; both are judgement and often differ.

Problem

"Delivery does not work"

The tester writes in the defect tracker: "Delivery does not work. Urgent!" The developer tries a cart of 30 € and sees 4.90 €. Tries 70 € and sees 0.00 €. All fine. They close the report as "cannot reproduce".

A week later a customer complains that they paid for delivery on an order of exactly 50.00 €. The tester saw it first. But their report gave neither the amount, nor what they expected, nor what they got.

The idea

A report is a letter to someone who was not there

A defect report has one job: someone else repeats the bug the first time and understands why it is a bug. So a good report has:

• a title that says the bug: "Delivery is 4.90 € for an order of exactly 50.00 €", not "Delivery does not work";
• steps: the shortest path to the bug, with the exact data. Not one step too many;
• expected: what the rule says. Free delivery;
• actual: what the system did. 4.90 €;
• environment: version, browser, device, when they matter;
• severity and priority (the next cards);
• a link to the requirement and to the test case that caught it.

One thing a report does not contain: who is to blame. The defect is in the product, not in the person.

More precisely, if you are curious

CTFL v4.0.1, section 5.5, gives the typical content of a report: a unique identifier, a title with a short summary, the date, author and organisation, the test object and environment, the context (test case, activity, technique), a description of the failure to enable reproduction and resolution, with steps, logs and screenshots, the expected and actual results, severity, priority, status and references. The syllabus also gives its objectives: information for the fix, a way to track the quality of the product, and ideas for improving the process.

The idea

The life of a defect

A report does not end when it is written. It goes through several states:

1. New: the tester writes it.
2. Open: at triage the team accepts it as a defect and gives it a priority.
3. In progress: the developer finds the cause and fixes it.
4. Fixed: the fix is in the version for testing.
5. Verified: the confirmation test passes, and so do the regression tests around it.
6. Closed.

The detours matter just as much. Rejected: not a defect, the system does what the rule says (then maybe the rule is wrong, and that is a question for the analyst). Duplicate: it is already reported. Deferred: it is a defect, but it will not be fixed now. Reopened: the confirmation test fails again.

More precisely, if you are curious

The names of the states differ between teams and tools (Jira, Azure DevOps, GitHub Issues). CTFL does not prescribe particular names: what matters is that the defect process is agreed, and everyone knows what each state means and who changes it.

Try it

Order the path

Order the states of the defect from the report to closing.

1. Verified: the confirmation test passes, and so do the regression tests↑↓
2. Closed↑↓
3. In progress: the developer finds the cause and fixes it↑↓
4. Open: the triage accepts it as a defect and gives it a priority↑↓
5. New: the tester writes the report↑↓
6. Fixed: the fix is in the version for testing↑↓

Check

The idea

Severity and priority are not the same

The two get mixed up all the time:

• Severity is how much harm the defect does: to the customer, to the money, to the data. The tester usually proposes it, because they saw the harm. Scale: critical, high, medium, low.
• Priority is how soon to fix it. The business or the product owner decides it, because it depends on the plan, the customers, the advertising. Scale: high, medium, low.

They often differ. A spelling mistake in the shop's name does little harm but is fixed today. A wrong annual report does a lot of harm but has 10 months until the next run.

Both are judgement, not fact. Two good testers may give a different severity, and that is normal. What matters is that you can say why.

Try it

Severity or priority?

1. There is a spelling mistake in the shop's name on the home page.

Low severity, low priorityLow severity, high priorityHigh severity, high priority

2. Accounting's annual report gives a wrong total of discounts. It runs once a year, next time in 10 months.

Low severity, high priorityHigh severity, low priorityLow severity, low priority

3. Nobody can pay by card.

Critical severity, high priorityMedium severity, medium priorityHigh severity, low priority

4. The hint of the phone field on the Bulgarian page is still in English.

Critical severity, low priorityHigh severity, high priorityLow severity, low priority

Check

Try it

The first report

In the tasks below the shop is not quite right. Each has one hidden bug, deliberately put into an otherwise correct system. In the Gherkin module such a version with one bug is called a mutant: your scenarios caught it if they failed on it. Here the bug has already been caught. You know roughly where it is. Your job is the report.

How the task works:

• you build the steps from the feature's step vocabulary and type the values;
• Try runs your steps on the system with the bug and shows what it gives;
• you choose expected (what the rule says) and actual (what you saw);
• you give a severity and a priority.

The check runs your steps on the system with the bug (the bug must show there) and on the correct one (it must not). The title is only partly checked: it must not be empty or "does not work". Severity and priority are compared with the model report, which you see after your attempt.

Find an amount at which the bug shows, and write the report.

The bug you caught: Free delivery

for orders well under 50 €

The rule: Delivery is free for an order from 50 € inclusive, otherwise it is 4.90 €. An empty cart cannot be ordered.

Title

Steps to reproduce

1. No steps yet. Add them from the vocabulary below.

The steps come from the vocabulary of the feature, as in the scenarios. The values go into the step itself: an amount like 50.00.

+ Given a cart of {amount} €+ Given an empty cart+ When I place the order

Try it on the system with the bug

Expected (what should happen by the rule)

Then delivery is … €Then delivery is freeThen the order is refusedThen the order is accepted

Actual (what happens in the system with the bug)

Then delivery is … €Then delivery is freeThen the order is refusedThen the order is accepted

Severity (how much harm the defect does)

criticalhighmediumlow

Priority (how soon to fix it)

highmediumlow

Check

The idea

How to find the shortest steps

You saw the bug after 12 steps: add, remove, change the quantity, enter a code, remove the code. The developer should not have to go through all of them. Before you write, narrow it down:

• remove a step and try again. Is the bug still there? Then the step is not needed;
• change one value at a time. At 30 € the bug is there, at 3 € it is not: so it is in the total, and you know roughly where the boundary is;
• write down the exact data. Not "some amount" but 30.00 €. Not "a few items" but 8 and then 3.

That is also the difference between "works on my machine" and "fixed the first time".

Exercise

Defect reports

1. The delivery fee. The bug is in the price of paid delivery.

Write a report from which the bug gets fixed the first time.

The bug you caught: Free delivery

in the price of paid delivery

The rule: Delivery is free for an order from 50 € inclusive, otherwise it is 4.90 €. An empty cart cannot be ordered.

Title

Steps to reproduce

1. No steps yet. Add them from the vocabulary below.

The steps come from the vocabulary of the feature, as in the scenarios. The values go into the step itself: an amount like 50.00.

+ Given a cart of {amount} €+ Given an empty cart+ When I place the order

Try it on the system with the bug

Expected (what should happen by the rule)

Then delivery is … €Then delivery is freeThen the order is refusedThen the order is accepted

Actual (what happens in the system with the bug)

Then delivery is … €Then delivery is freeThen the order is refusedThen the order is accepted

Severity (how much harm the defect does)

criticalhighmediumlow

Priority (how soon to fix it)

highmediumlow

Check

2. The eleventh item. The bug is around the upper limit of the cart.

Write a report from which the bug gets fixed the first time.

The bug you caught: At most 10 of a product

when the cart goes past the upper limit

The rule: The cart can hold at most 10 of one product. Adding beyond 10 is refused with a message, and the cart stays the same. You add at least 1 item.

Title

Steps to reproduce

1. No steps yet. Add them from the vocabulary below.

The steps come from the vocabulary of the feature, as in the scenarios. The values go into the step itself: an amount like 50.00.

+ Given an empty cart+ Given the cart has {count} of the product+ When I add {count} more

Try it on the system with the bug

Expected (what should happen by the rule)

Then the cart has … of the productThen adding is refusedThen adding is accepted

Actual (what happens in the system with the bug)

Then the cart has … of the productThen adding is refusedThen adding is accepted

Severity (how much harm the defect does)

criticalhighmediumlow

Priority (how soon to fix it)

highmediumlow

Check

3. The refusal that empties. The bug is in what stays in the cart after a refusal.

Write a report from which the bug gets fixed the first time.

The bug you caught: At most 10 of a product

in what stays in the cart after a refusal

The rule: The cart can hold at most 10 of one product. Adding beyond 10 is refused with a message, and the cart stays the same. You add at least 1 item.

Title

Steps to reproduce

1. No steps yet. Add them from the vocabulary below.

The steps come from the vocabulary of the feature, as in the scenarios. The values go into the step itself: an amount like 50.00.

+ Given an empty cart+ Given the cart has {count} of the product+ When I add {count} more

Try it on the system with the bug

Expected (what should happen by the rule)

Then the cart has … of the productThen adding is refusedThen adding is accepted

Actual (what happens in the system with the bug)

Then the cart has … of the productThen adding is refusedThen adding is accepted

Severity (how much harm the defect does)

criticalhighmediumlow

Priority (how soon to fix it)

highmediumlow

Check

Test

Check yourself

1. What must the person reading the report be able to do?

Repeat the bug the first time, without asking the authorWork out who is to blameFind the code that is wrong

2. Who sets the priority?

Usually the business or the product owner, together with QA and the developersAlways the tester aloneThe developer who will fix it

3. "Expected" in the report comes from…

what the tester thinks is rightthe previous versionthe rule, the requirement or the acceptance criterion

4. The developer rejects the report with "works on my machine". What was most likely missing?

A screenshotThe exact steps, data and environmentA higher priority

5. Why does the root cause of a defect matter once it is fixed?

To know who made the mistakeSo that the same kind of mistake does not happen again: for example a missing rule for the boundaryIt does not matter

Check

Start the lesson over

Related lessons

• Builds on: Why programs fail · Test levels · Test types · The plan and risk-based testing
• Continues in: Tests in CI · Promo codes: from the request to the release
• Tasks on this topic: Levels, types and process