Why and what to automate

In short: An automated test is a program that runs the system and compares the result, with no person involved. It pays off for rules that rarely change, are checked on every change and cost money when they break. It does not pay off for a one-off check, for a screen that still changes, or for anything that needs human judgment. Its real cost is maintenance.

Problem

Two days of manual tests every week

The shop releases a new version every Friday. Before every release QA goes through 120 test cases by hand: delivery, the cart, promo codes, returns. It takes two days.

Last week there was a public holiday, and there was time for 80. The 40 that were skipped were "the old ones, they do not break". On Monday a customer paid for delivery on an order of exactly 50.00 €. The change was in the promo code logic, and it broke the delivery boundary.

The idea

Test automation

Test automation is a program that runs the system, gives it the input from the test case and compares the result with the expected one. No person involved, and the same on every run.

What it gives you:

regression becomes cheap. 120 cases run in minutes, on every change, not over two days once a week;
the same every time. A program does not get tired at case 80 and does not skip "the old ones";
fast feedback. The programmer learns that the boundary is broken five minutes later, while they still remember what they touched;
living documentation runs. The scenarios from the Gherkin module stop being text in a folder: they become a check that passes or fails.

Automation does not invent tests. A person still writes the test case. Automation repeats it.

The idea

What is worth it and what is not

An automated test is paid for once when it is written and then on every change. So it is worth it when it runs many times and changes rarely.

It is worth it for:

rules that do not change every week: delivery, the cart limit, discounts;
checks before every release (regression);
many examples of the same rule: ten amounts around the boundary;
places where a bug costs money or trust.

It is not worth it (or not yet) for:

a one-off check: a data migration that happens once;
a screen the designer still changes every day;
anything that needs judgment: is the form clear, does it look right, is it convenient. That is usability, and people look at it;
exploratory testing: there the question is invented while you test.

Try it

Automate or not?

1. The free delivery rule: it is checked before every release, and there is a release every week.Keep it manual: it is simpleAutomate: the rule is stable, it is checked often and a bug in it costs moneyDo not test it: it has been tested already

This is exactly what automation is for: the same thing, many times, with a clear expected result.

2. The new home page: the designer changes it every day this week.Automate right away so that it does not breakAutomate every versionManually for now: an automated test would have to be rewritten every day

A test of something that still changes breaks because of the change, not because of a defect. Automate it once the screen settles.

3. Do older customers understand the returns form?Automate it with a UI testManually, with real people: this is judgment, not comparing with an expected resultAutomate it with a component test

A program compares a result with an expected one. "Do they understand it" has no expected result that can be written down in advance.

4. Migrating the old orders to the new database: it happens once, on Saturday night.A check with a script just for the migration, which does not join the permanent testsGherkin scenarios for every orderNo check at all

The check is needed, but it is a one-off. The script is written for the day of the migration and then thrown away.

The idea

The risks

Automation often fails not because of the tool but because of the expectations:

"It will find all the defects." A test checks only what someone thought of. If nobody thought of the empty cart, automation will not check it, however many times it runs. The mutants in the Gherkin module slipped through in exactly this way.
Underestimated maintenance. Tests are code. When the threshold becomes 60 €, the scenarios, the steps and the tests change. If nobody maintains them, a month later half of them are red and the team stops looking.
A test nobody trusts. If a test fails "sometimes", people start running it again until it passes. Then a real failure slips through too.
Everything through the screen. Tests through the browser are slow and break when a button moves. Most checks belong further down.
The tool as the goal. "We bought a tool" is not a strategy. First comes what has to be checked, then what to check it with.

More precisely, if you are curious

ISTQB CTFL v4.0.1 covers tools in chapter 6. Section 6.2 lists the benefits, among them less repetitive manual work, more consistency, faster feedback and more time for new tests, and the risks, among them unrealistic expectations, inaccurate estimates of time and cost (including maintenance), using a tool where manual testing is more suitable, relying on a tool too much and ignoring human critical thinking, and dependence on the tool vendor or on open-source software that may be abandoned.

For those who will write automation for a living, ISTQB has a separate advanced certificate: CTAL-TAE (Test Automation Engineering). It is about the architecture of automation and is not part of this course.

The idea

Which level for which check

The pyramid from the module on levels says how many tests on which level. In automation it becomes concrete:

scenarios (the next lesson) check the rule in the words of the business. They are the living documentation;
component tests check one piece of code, fast and with many examples: ten amounts around the boundary in milliseconds;
API tests send real requests to the shop's server, without a browser;
UI tests open a browser and click like the customer. Only a few, for the paths that must not break.

In this module the rule stays the same throughout: delivery is free from 50.00 € inclusive, otherwise it is 4.90 €, and an empty cart is refused. You will see it automated on every level, in Java, C#, Python and JavaScript, and you will see which test catches which bug.

Here you need to be able to read code in one of the four languages. What you need is in the programming course, up to the module "Real-world programs": classes, exceptions and component tests.

Try it

The path of one rule

Put the path of the delivery rule in order, from the agreement to the report.

The rule is checked with component tests too
The scenario is connected to the code through steps
The tests run on every change to the code
The rule is agreed and written down as a scenario
The report says what failed and why

This is also the plan of the module: the steps, the component tests, the API and the screen, then CI and the report. The scenario comes first because it is the agreed rule. The test code comes after it.

Exercise

An automation plan

1. What first and why. The shop is just starting with automation.

1. Which of these four is best to automate first?The colour of the "Place order" buttonA check that the new supplier has signed the contractThe promo code discount: a stable rule, many examples, money

The discount is checked on every change and a bug in it is a loss. The colour is checked with your eyes, and the contract is not in the program at all.

2. The team automates 300 tests in a week. A month later half of them are red and nobody looks at them. Which risk has come true?The tool is badUnderestimated maintenance: tests are code and have to be looked after like codeThere are too few tests

A test nobody trusts is worse than a missing one: it makes noise and hides the real failures.

3. The manager says: "We have automation, we do not need testers any more." What are they missing?That automation is slowerAn automated test checks only what someone thought of; new questions, judgment and exploration stay with peopleNothing, he is right

Automation frees up the time spent on repetition. That time goes to the tests nobody has thought of yet.

Test

Check yourself

1. What is an automated test?A test the system writes by itselfA program that runs the system and compares the result with the expected one, with no person involvedAny test recorded in a tool

A person writes the test case. Automation runs it and compares.

2. What makes a rule a good candidate for automation?It is stable, it is checked often and a bug in it is expensiveIt is checked onceIt is new and still being discussed

Automation pays for itself through repetition. Without repetition its cost does not come back.

3. What is the biggest ongoing cost of automation?The tool licenceMaintenance: tests change with every change to the rule or the screenThe electricity for the server

A test is written once and maintained for years. That is why you automate what does not change every day.

4. Where in the pyramid do most automated tests go?At the top: everything through the screen, as the customer sees itEqually on every levelAt the bottom: many fast component tests, fewer through the API, few through the screen

At the bottom tests are fast and fail exactly where the bug is. At the top they are slow and break on small changes.

✓ You finished the lesson.From scenario to steps →

Related lessons

Section titled "Related lessons"

Builds on: Test levels · Test types · Testing in the life cycle · Usability and accessibility
Continues in: From scenario to steps · Component tests · API and user interface · Tests in CI
Tasks on this topic: Automation

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·

Cookies

We use Google Analytics to see which lessons get read and where people give up. No ads. Nothing is counted unless you accept.

What exactly

Google Analytics 4 through Google Tag Manager (Google Ireland Ltd.).
Cookies _ga and _ga_* for up to 2 years: a random browser id and when it was here.
It records the pages you open and what you do in lessons (cards, 3D scenes, whether an answer to an exercise is right), what you search for on the site, how fast the page loads, your approximate country, device and browser. Never your name, email or the code you write.
Your lesson progress stays in your browser only and needs no consent.