Measuring after the release

In short: A released solution is a hypothesis: "if we give 10 % with a code, more new customers will come". Whether it is true becomes clear after the release, with a metric agreed before it: what is measured, what it was, what it should become and by when. Then the result is compared honestly and leads to a decision: it stays, it changes or it stops.

Problem

Two thousand uses. Is that a success?

The promo codes were released three months ago. Marketing reports: "The code was used 2,000 times!"

The owner asks: "Did new customers come? Or did the people who buy anyway just get 10 %?" Nobody knows. Nobody recorded how many new customers there were before the campaign. Nobody said what "success" means.

The idea

Solution evaluation

Acceptance said "we can do our work". It did not say whether the solution delivers the value it was requested for. The promo codes were not requested for the sake of codes. They were requested because of a need: more new customers in the autumn.

Every released solution is a hypothesis: "if we give 10 % with a code, more new customers will come". Whether it is true becomes clear after the release. Solution evaluation is measuring after the release: does the solution deliver the value, and if not, why and what to do.

This is the last step of validation. Verification asks "did we build it right", acceptance "can we work with it", and evaluation "was it worth it".

More precisely, if you are curious

BABOK Guide v3 has a whole knowledge area, Solution Evaluation, with five tasks: Measure Solution Performance, Analyze Performance Measures, Assess Solution Limitations, Assess Enterprise Limitations, and Recommend Actions to Increase Solution Value. It also applies to solutions that are not fully released yet: a pilot, a beta, a first part.

The idea

The value metric

To check the hypothesis you need a value metric, agreed before the release. It has five parts:

the need it comes from: more new customers in the autumn;
what is measured: the number of first orders from new customers per month. Not "code uses": they grow even when only existing customers get a discount;
the baseline: how many there were without the change. For example, an average of 400 a month last autumn;
the target: how many there should be. 500 a month, that is 25 % more;
the deadline: by when. By the end of November.

It is good to have a limit that must not be crossed too: for example, the average order value must not drop below 35 €. Otherwise the target can be met with orders the shop loses money on.

A metric is a requirement like any other: checkable, with a number and a deadline. "The campaign has to be successful" is not a metric.

Try it

Is it a good metric?

1. The need: "more new customers in the autumn". Which metric is best?The number of first orders from new customers per month, compared with the same months last yearThe number of visits to the home pageThe number of times the code was used

Code uses grow even when existing customers just get a discount. The metric has to be about the need, not about the feature.

2. "The campaign has to be successful." What is missing to make it a metric?NothingWhat is measured, what it is now, what it should become and by whenThe name of the campaign

"Successful" cannot be checked. Just like a requirement from the requirements module: without a number and a deadline nobody can say whether it is met.

3. Why is the baseline recorded before the release?It is not neededTo measure fasterAfter the release there is no way to find out what it was without the change

Without "before" there is no "more". The old data often gets deleted, lost or counted differently.

The idea

How to read the result

Three months later the numbers arrive. How to read them honestly:

compare with the baseline, not with the hopes. 440 new customers a month is 10 % growth against a target of 25 %;
other causes. If a competitor closed the same month, the growth is not only because of the code. A comparison with a group without the change (customers who were not shown the code), or with the same period in another year, helps;
where the cost goes. Who uses the code? If most are existing customers, the shop pays 10 % for orders it would have had anyway;
the limitations. Sometimes the solution does what it should, but something prevents the value from being taken. In the solution: marketing cannot launch a new code without a programmer. In the organisation around it: the warehouse cannot keep up with the orders in the first week.

The idea

The recommendation: it stays, changes or stops

Evaluation ends with a recommendation:

it stays, if it delivers the value;
it changes, if it works, but weakly or in the wrong place. Here: the code is valid only for the first order of a new customer;
it stops, if the cost is greater than the benefit and a change will not help.

A change is a new requirement. It goes along the whole path of this course: the need, the story, the examples, the scenarios, the matrix, the tests, acceptance, and measuring again. So the path from the request to the release is not a straight line but a cycle.

Which of the three it is, the owner decides. The analyst and QA give the numbers and say honestly what they show and what they do not.

Try it

The path of the metric

Put the steps from the need to the decision after the release in order.

Measuring and comparing with the baseline
The baseline: how many there were last autumn
The need: more new customers in the autumn
The decision: it stays, it changes or it stops
The metric: first orders from new customers per month
The target and the deadline: how many more and by which month
Releasing the promo codes

The metric, the baseline and the target come before the release. If they are chosen after it, the ones that look good get chosen.

Exercise

The promo codes after three months

1. How you read the numbers. The shop measures the promo codes after the autumn.

1. New customers grew by 8 % against a target of 25 %. But 70 % of the code uses come from existing customers who buy anyway. What do you recommend?A change: the code is valid only for a first order, and a new measurementStop it at once without finding out whyThe campaign is a success, it continues as it is

The solution works weakly, and its cost goes elsewhere. The recommendation is a change of requirement, which goes through everything again: story, scenarios, tests.

2. New customers grew by 30 %. The same month a competitor closed down. What do you say?The target is metThe metric is wrongThe growth cannot be attributed to the code alone: a comparison is needed that removes the other cause

A metric says what happened, not why. A comparison with customers who did not get the code, or with another period, helps separate the effect.

3. Marketing cannot launch a new code without a programmer, and every campaign is two weeks late. What is this?A defectA solution limitation: it does what was asked, but it does not let the value be taken in timeAn enterprise limitation

The solution is correct, but it gets in the way of the value. That is a reason for a new story: marketing managing the codes.

4. The codes work, but the warehouse cannot keep up with the orders in the first week of the campaign. What is this?A defect in the discount codeA solution limitationA limitation of the organisation outside the solution: the people and the process around it

The problem is not in the software. The value is lost because of the warehouse capacity, and the fix is there.

Test

Check yourself

1. What does solution evaluation after the release ask?Are there defects?Does the solution deliver the value it was requested for?Did the tests pass?

This is validation after the release. A solution without defects may deliver no value at all.

2. What is part of a good value metric?Only the targetA baseline, a target and a deadlineThe number of features

Without a baseline there is no comparison, without a deadline there is no time to check.

3. When is the metric agreed?Never, the numbers speak for themselvesAfter the release, once the numbers are inBefore the release, together with the need

A metric chosen after the numbers always shows success: the one that looks good gets picked.

4. The result is below the target. What comes next?NothingFind out why and recommend a change or a stop; a change is a new requirementLower the target

Evaluation ends with a recommendation. That turns the path from the need to the release into a cycle.

✓ You finished the lesson.Promo codes: from the request to the release →

Related lessons

Section titled "Related lessons"

Builds on: What a story is · Acceptance criteria · When a requirement changes · Acceptance testing with the business
Continues in: Promo codes: from the request to the release
Tasks on this topic: Validating with the client

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·

Cookies

We use Google Analytics to see which lessons get read and where people give up. No ads. Nothing is counted unless you accept.

What exactly

Google Analytics 4 through Google Tag Manager (Google Ireland Ltd.).
Cookies _ga and _ga_* for up to 2 years: a random browser id and when it was here.
It records the pages you open and what you do in lessons (cards, 3D scenes, whether an answer to an exercise is right), what you search for on the site, how fast the page loads, your approximate country, device and browser. Never your name, email or the code you write.
Your lesson progress stays in your browser only and needs no consent.