Performance and load
In short: A performance requirement says which action, how fast, for what share of requests and under what load: "95% of cart openings are under 300 ms with 200 customers at the same time". A load test checks what was promised, stress looks for where it breaks, and endurance whether it lasts for hours.
Problem
It was fast while I shopped alone
The requirement is "The shop must be fast". The tester opens the cart: 0.8 seconds. Fast. The requirement is met.
On Black Friday 600 people enter the shop at the same time. The cart opens in 14 seconds, payments time out, and customers fill their carts again and again, which makes everything even slower. The shop loses its best day of the year.
The tester's test was right. The requirement said nothing that could fail.
The idea
The five parts of the requirement
A performance requirement says five things:
- Which action: opening the cart, search, payment. Not "the shop".
- What measure: response time, orders per minute, error rate.
- What limit: under 300 ms, at least 50 orders a minute, under 1% errors.
- For what share: 95% of requests, not the average.
- Under what load and conditions: 200 customers at the same time, in an environment like the real one, with all 12,000 products.
Together: "95% of cart openings are under 300 ms with 200 customers at the same time". This sentence can fail. "It must be fast" cannot.
More precisely, if you are curious
In ISO/IEC 25010:2023 this is the characteristic performance efficiency, with three sub-characteristics: time behaviour, resource utilisation and capacity. How such a requirement is written as an architecture scenario, and why availability percentages multiply along a chain of services, is in the lesson Non-functional requirements you can measure of the software engineering course.
The idea
Why not the average
Ten customers open the cart. Nine wait up to 0.6 seconds, one waits 9.8. The average is 1.3 seconds and looks fine. But one customer waited almost 10 seconds, and the average hides it.
A percentile says how many requests are under the limit. p95 = 300 ms means: 95% of requests are under 300 ms, and 5% may be slower. The requirement says exactly how many slow requests are acceptable, and does not let a few very fast ones hide the slow ones.
For payment, where every slow customer is a lost order, teams often ask for p99 too.
Try it
Work it out
The idea
Three questions, three kinds of tests
Performance tests are non-functional testing, usually at system level. They answer different questions:
- Load testing: does the system handle what is expected? 200 customers, as many as at the peak. It checks the requirement.
- Stress testing: where does it break, and how? Customers grow beyond what was promised until something stops. Then: does the shop recover on its own, are orders lost?
- Endurance (soak) testing: does it last? 48 hours at the usual load. It catches what builds up slowly: memory, open connections.
There are also spikes (spike testing): from 80 to 600 customers within a minute, when the sale email goes out.
For all of this there are tools that run thousands of fake customers. We do not teach them here: a tool is useless if the requirement does not say what to measure and when to fail.
More precisely, if you are curious
Scalability and capacity are checked with similar tests: how many customers the system takes, and how that grows when a server is added. In ISO/IEC 25010:2023 scalability is part of flexibility, and capacity is part of performance efficiency.
Try it
What kind of test?
The idea
Who gives the numbers
The numbers are not the developers' ("the server takes 150") and not the tester's. They are a business need: how many customers came to the last sale, how many will come with the new advertising, how many seconds of waiting mean a lost order.
The analyst asks for them the way they ask for rules: with an example. "Last Black Friday there were 420 people in 10 minutes at the peak. This year we expect 50% more. How much waiting at payment is acceptable?"
Then the requirement is accepted like any other: a test under the agreed conditions, and the result compared with the agreed limit. If it passes but customers still complain, the requirement is wrong, not the program. That is validation of the non-functional.
More precisely, if you are curious
CT-AcT v1.0, chapter 4, treats performance as part of acceptance testing: the measures are agreed with the business beforehand, and the test environment has to be representative. Performance tests are expensive (environment, data, time), so they are a good candidate for selection by risk: payment and the cart first, the "About us" page later.
Exercise
A requirement for the sale
1. Black Friday. Review marketing's requirement.
Mark the words that make the requirement unverifiable or ambiguous, and choose the kind of each problem.
Press the words that are a problem; press again to unmark. The arrow keys move from word to word.
The problems in the text
- “fast” unverifiable: What should be fast, and how fast? Every action that is measured gets its own limit: opening the cart, search, payment.
- “average time” ambiguous: The average hides the slow customers: 90 fast openings and 10 of 15 seconds give a nice average. The requirement is written with a percentile: 95% of openings are under 2 seconds.
- “many users” unverifiable: How many? The measure is a number from the expected load: 200 customers at the same time, 50 orders a minute at the peak.
- “must not be slow” unverifiable: A negation without a limit: there is no value at which the test fails. For example: 95% of payments complete in under 4 seconds, not counting the bank's time.
The sentence with the average time looks measurable, because it has a number. But the average hides the slow customers. With a percentile and a load ("95% of openings under 2 seconds with 200 customers") it becomes a good requirement. The other three have neither a limit nor a load.
Test