Skip to content
wannacode

Non-functional requirements you can measure

In short: A non-functional requirement says how well a system works (speed, availability) and is written as a scenario with a measure, such as p95 and p99, not as "it should be fast".

Problem

The dashboard is green, and the customers wait

The owner of the shop tells the team: "The shop must be fast and never go down." The team puts the average payment time on the dashboard with the goal "under 500 ms". Here are the last 20 payments from the log:

// 20 payments from the log, in milliseconds
int[] times = [120, 95, 130, 110, 105, 2400, 140, 98, 115, 125,
    101, 135, 990, 108, 122, 99, 118, 1500, 112, 104];
int average = times.Sum() / times.Length;
Console.WriteLine($"average: {average} ms");
Console.WriteLine("goal: under 500 ms " + (average < 500 ? "✓" : "✗"));
Output
average: 341 ms
goal: under 500 ms ✓

The dashboard is green. And support gets complaints: "I pressed 'Pay' and waited two seconds. I pressed it again." Both the dashboard and the customers are telling the truth.

"Fast" does not say fast for whom, under what load and measured how. "Never down" does not say how many minutes a month are acceptable. A requirement like that cannot be checked, so it cannot be met either.

The idea

The architect's work starts with numbers

This module is about the real work of an architect: decisions that are expensive to change later, and how to defend them with numbers instead of taste. At the end your first architecture task waits for you: the shop launches in three countries and opens to outside sellers. You turn that into measurable requirements, propose at least two options with their cost, record the decision in an ADR and plan the migration of the live system so that every step can be undone.

The first skill is here. A functional requirement says what the system does: "the customer pays by card". A non-functional requirement (NFR) says how well: how fast, how often it works, how many customers it handles, how much it costs. Architecture is decided mostly by the second kind: the same function can be one server or three countries with replicas.

An NFR you can measure is written as a quality scenario:

  • source: a customer in the mobile app;
  • stimulus: presses "Pay";
  • environment: a normal day, up to 50 payments a second;
  • artifact: the order chain;
  • response: the order is paid, or the customer sees a clear error;
  • measure: 95% of payments answer within 800 ms and 99% within 2 s, measured on the server over the last 7 days.

The numbers here are an example: the real ones come from the business and from what the system can do today. What matters is that after a week anyone can say "met" or "not met".

The idea

Percentiles instead of the average

Sort the times from the fastest to the slowest. The percentile p95 is the time below which 95% of requests fall: only 5 in 100 are slower. p50 is the middle (the median), and p99 is the time one customer in a hundred waits.

Here is how to find it with the nearest rank: of n sorted times, the p-th percentile is the number in place ⌈p · n / 100⌉. Integer division with +99 rounds up without fractions:

// nearest rank: ceil(p · n / 100), in integers
static int Percentile(int[] sorted, int p)
{
    int rank = (p * sorted.Length + 99) / 100;
    return sorted[rank - 1];
}

The same 20 payments:

int[] sorted = times.Order().ToArray();
foreach (int p in new[] { 50, 95, 99 })
    Console.WriteLine($"p{p}: {Percentile(sorted, p)} ms");
Output
p50: 115 ms
p95: 1500 ms
p99: 2400 ms

Half the customers wait up to 115 ms. But p95 is 1.5 seconds: one in twenty waits at least that long. The average of 341 ms describes no customer at all: the fast ones are below it and the slow ones far above. That is why speed requirements are written with percentiles, and the average is left for other things, such as how much CPU time you pay for.

The idea

A latency budget along the chain

A payment is not one program. The request goes through the gateway, the orders service, the external payment provider and the warehouse. If the goal is p95 within 800 ms, every part gets its share:

  • gateway: 50 ms;
  • orders: 150 ms;
  • payment (external provider): 450 ms;
  • warehouse: 100 ms;
  • reserve: 50 ms.

This is a latency budget. Every team knows its number, and when the warehouse asks for 300 ms, the conversation is about figures: where do the 200 ms come from? The numbers are an example again.

There is a trap too. When a page calls several services at once and waits for all of them, it is slow as soon as at least one is slow. The more services, the more often that happens.

Try it

Predict the output

Each service is slow for 1% of requests. The page calls 1, 5 or 20 such services at once and waits for all of them. What share of the pages is slow?

// each service is slow for 1% of requests (above its p99)
foreach (int n in new[] { 1, 5, 20 })
{
    double slow = 1 - Math.Pow(0.99, n);
    Console.WriteLine($"services: {n} → slow pages: {slow * 100:F2}%");
}
Answer

The idea

Availability: the percentages multiply

Availability is the share of time in which the system answers correctly. It is usually said in "nines". Here is how much downtime each allows in 30 days (30 × 24 × 60 = 43,200 minutes):

  • 99%: 432 minutes, a little over 7 hours;
  • 99.9%: 43.2 minutes;
  • 99.99%: 4.3 minutes.

An order succeeds only if the gateway, the orders service, the payment and the warehouse all work. If every service fails independently of the others, the probability that all of them work is the product of their percentages. That is why the chain is weaker than its weakest service.

See it

The chain of services

static double Chain(List<Service> services)
{
    double total = 1.0;
    foreach (var s in services)
    {
        total *= s.Availability / 100;
        var line = $"{s.Name,-9} {s.Availability:F2}%";
        Console.WriteLine($"{line} → {total * 100:F2}%");
    }
    return total;
}


// example numbers, not the promise of a real provider
List<Service> services =
[
    new("gateway", 99.95),
    new("orders", 99.9),
    new("payment", 99.5),
    new("warehouse", 99.9),
];
double total = Chain(services);
long down = (long)Math.Round((1 - total) * 30 * 24 * 60);
Console.WriteLine($"downtime in 30 days: {down} min");
Step by step, in words
  1. An order goes through four services. It is placed only if every one of them works at that moment. The percentages are examples.
  2. gateway: 99.95%. The chain starts at 100% and after the first service it is 99.95%.
  3. orders: 99.90%. The chain so far: 99.95% × 99.90% = 99.85%.
  4. payment: 99.50%. The chain so far: 99.85% × 99.50% = 99.35%. The external payment provider is the weakest link.
  5. warehouse: 99.90%. The chain so far: 99.35% × 99.90% = 99.25%.
  6. The chain: 99.25%, lower than its weakest service. In 30 days that is 323 minutes when nobody can order.

A service is described by a name and an availability:

record Service(string Name, double Availability);
static double Chain(List<Service> services)
{
    double total = 1.0;
    foreach (var s in services)
    {
        total *= s.Availability / 100;
        var line = $"{s.Name,-9} {s.Availability:F2}%";
        Console.WriteLine($"{line} → {total * 100:F2}%");
    }
    return total;
}


// example numbers, not the promise of a real provider
List<Service> services =
[
    new("gateway", 99.95),
    new("orders", 99.9),
    new("payment", 99.5),
    new("warehouse", 99.9),
];
double total = Chain(services);
long down = (long)Math.Round((1 - total) * 30 * 24 * 60);
Console.WriteLine($"downtime in 30 days: {down} min");
Output
gateway   99.95% → 99.95%
orders    99.90% → 99.85%
payment   99.50% → 99.35%
warehouse 99.90% → 99.25%
downtime in 30 days: 323 min

Every service is 99.5% or more, but the order is 99.25%: 323 minutes without orders in a month, more than five hours. Every new synchronous dependency lowers the number. That is why the warehouse should not be in the path of the payment: it can learn about the order later, from an event in a queue. Then a warehouse outage does not stop the payment.

:F2 writes the number according to the culture of the system. On a computer with a Bulgarian culture you would see 99,25%; the lesson's check runs with the invariant culture and a dot.

Try it

Fill in

The availability of the chain is the product of the availabilities. What is missing?

double total = 1.0;
foreach (var s in services)
    total  s.Availability / 100;

Try it

Find the bug

A colleague computes the availability of the chain like this and promises a client 99.5%, "as much as the weakest service". Which line has the bug?

Pick the line with the bug.

The idea

An error budget

Three terms you will meet:

  • SLI (service level indicator): what you measure, for example "the share of successful payments";
  • SLO (objective): the goal, for example "99.5% for the week";
  • SLA (agreement): a contract with a client, with a penalty if the goal is missed. The SLA is usually weaker than the SLO, to leave a margin.

If the goal is 99.5%, the remaining 0.5% is the error budget. With 60,000 payments a week that is 300 failed ones. While there is budget, the team ships new versions. Once it is spent, only reliability fixes ship until the end of the period. That turns the argument "new things fast, or stable" into arithmetic instead of opinion.

The idea

When it hurts

  • "99.99% for everything". Every further nine asks for more copies, on-call duty and a more complex system. The weekly report for accounting does not need what the payment needs. Set a high goal only where money or trust require it.
  • Requirements nobody measures. A list of 40 NFRs without measurement is noise. Better five, each with a measure and a person who watches it.
  • A percentile from little data. From 20 times, p99 is only the slowest one. You need thousands of measurements.
  • Requirements that fight each other. More security means more checks, and checks take time. Stronger consistency means lower availability. That is the next lesson: trade-offs.

The measuring itself (metrics, dashboards, alerts) is infrastructure and is taught in the DevOps course ZeroToOps. Here is the part the architect decides: what is measured and how much is enough.

More precisely, if you are curious

Percentiles are not averaged. The p95 of two servers is not the average of the two p95s. Metrics systems keep a histogram (the number of requests in each time interval) and compute the percentile from the sum: HdrHistogram, the histograms in Prometheus, OpenTelemetry. Percentiles along a chain do not add up exactly either: the p95 of the sum is not the sum of the p95s. The latency budget is a plan, not a formula.

Coordinated omission. If a load tool waits for the answer before sending the next request, it does not measure the times when the system was jammed. Then p99 looks better than the truth.

Independence. The multiplication assumes the services fail independently. If all of them are in one data centre and the power goes, they fail together. The other way round, two copies of one service that fail independently give 1 − (1 − a)²: two copies at 99% each are 99.99%. That is why copies sit in different zones.

Where the words come from. The quality scenario comes from the methods of the SEI (Software Engineering Institute), such as ATAM. ISO/IEC 25010 lists the characteristics of quality: performance, reliability, security, maintainability and others. SLI, SLO and the error budget come from Google's book on SRE.

Exercise

An error budget for one week

Solve the task in your own editor.

The goal for payments is 99.5% successful. The plan is 60,000 payments for the week. The failed ones by day are 20, 35, 150, 60 and 50. Write Budget, which computes how many errors the goal allows, subtracts the failed ones day by day and finally says whether there is budget for new versions.

Output
budget: 300 errors out of 60000 requests
Mon: 20 → left 280
Tue: 35 → left 245
Wed: 150 → left 95
Thu: 60 → left 35
Fri: 50 → left -15
the budget is spent: only fixes until the end of the week

More tasks

  • Speed. Add an SLO for speed: p95 within 800 ms. For every day you have 20 times. Which days break the goal?
  • Two copies. The payment has two providers that fail independently, at 99.5% each. If the second one takes over when the first one fails, what is the availability of the payment, and what does the chain become?

Test

Check yourself

1. Which of these is a requirement you can check?
2. The average time is 341 ms and p95 is 1500 ms. What does that mean?
3. An order goes through four services, each with 99.9% availability. What is the availability of the order?
4. The goal is 99.5% successful payments, and the error budget for the week ran out on Wednesday. What is usually done?
5. Why does p99 of 20 measurements tell you little?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·