Acceptance testing with the business
In short: Acceptance testing asks a different question: not "does the system do what is written" but "can I do my job with it". People from the business do it, with real tasks and realistic data, while QA prepares, observes and records. Findings are defects, change requests or questions. The later "this is not it" is heard, the more expensive it is; that is why a demo and a prototype show the direction earlier.
Problem
It works, but it is not what we need
The promo codes passed all the tests: 10 %, not valid on discounted products, one paid order, a validity period. The scenarios are green, and so are the tests in CI.
During acceptance testing the accountant does what she does every month: she reconciles the discounts. She opens an order and sees "Discount: 2.00 €". Which code? It does not say. She has no way to match the discounts with the campaigns. "Everything works, but I cannot do my job." The release is three days away.
The idea
User acceptance testing
All the tests so far asked "does the system do what is written". That is verification. User acceptance testing (UAT) asks something else: can I do my job with this system? That is validation.
That is why it is done by the people who will work with the system: the accountant, the marketing employee, the warehouse. Not QA and not the programmers. They do their real work, with tasks from their everyday job and realistic data, in an environment that looks like the real one.
QA is not redundant. QA prepares the environment and the data, helps derive the tasks from the work processes, observes and records the findings. But nobody except the business can give the verdict "this is enough for me".
The idea
How an acceptance goes
Preparation. Which people, which work processes, which environment, what data. And the entry criteria: the system tests have passed, there are no open critical defects. People from the business have little time, and it must not be spent on bugs the lower levels should have caught.
Execution. Everyone does their tasks: "launch an autumn campaign", "reconcile the discounts for October". QA watches without prompting and records where the person stops, what they look for, what surprises them.
Findings. There are three kinds, and each follows its own path:
- a defect: the system does not do what was agreed. It is reported and fixed;
- a change request: the system does what was agreed, but what was agreed is not enough. It goes through change control: assessment, priority, decision;
- a question: something nobody has decided. It is recorded, and there is someone to answer it.
Decision. Accepted, accepted with conditions, or not accepted. The owner decides, by the exit criteria agreed before the start.
More precisely, if you are curious
ISTQB CTFL v4.0.1 describes acceptance testing as one of the test levels and lists its main forms: user acceptance testing, operational, contractual and regulatory acceptance testing, and alpha and beta testing (the module "Levels, types and process"). ISTQB also has a separate certificate on acceptance testing, Certified Tester Acceptance Testing (CT-AcT). It is about the collaboration between product owners, business analysts and testers: how acceptance criteria and acceptance tests are written together with the business, and how acceptance is carried out.
Try it
What is this finding?
The idea
Why "this is not it" comes so late
The story with the accountant is a classic. The requirements were correct and complete for the discount. Nobody had asked accounting what they do with the discounts afterwards. The stakeholder nobody asks shows up at acceptance.
The causes are usually the same:
- the requirement was written as a solution ("a code field in the cart"), not as a need;
- the people who will work with the system were not at the meetings;
- nobody watched the work process, they only asked about it;
- the first feedback from the business comes only with the finished system.
The last one is the most expensive. At acceptance everything is built, and every change is rework. Three days before the release there are two bad choices left: release without what is needed, or postpone.
The idea
The demo and the prototype
The answer is for the feedback to come earlier, while a change is cheap:
- a prototype: a sketch or a page you can click through, with no real logic. The accountant sees the order screen before the first line of code and says: "where is my code?" It costs an hour;
- a demo: at the end of every sprint the team shows the working part to the people who will use it. It is best if they do not just watch but try a real task themselves;
- early access: a small group from the business works with the finished parts while the others are being built.
A demo and a prototype do not replace acceptance. They make sure acceptance holds no surprises, only confirmation. It is the same idea as shift-left: the check moves earlier, because it is cheaper there.
More precisely, if you are curious
In Scrum the demo happens in the sprint review: the team presents the results to the key stakeholders, and together they work out what to do next. The Scrum Guide calls the review a working session that should not be limited to a presentation.
Try it
From the cheapest to the most expensive feedback
Order the places where the business can say "this is not it", from the cheapest to the most expensive.
A paper sketch on the board with the storyA prototype you can click through, with no real codeA demo of a working part at the end of the sprintA customer complaint after the releaseAcceptance testing with real work before the release
A sketch costs minutes, a prototype hours, a demo a week of work. At acceptance everything is built, and after the release the bug is already with the customers.
Exercise
Accepting the promo codes
1. The plan. The promo codes are ready for acceptance.
Test