A/B test

In short: An A/B test compares two variants on randomly split visitors and checks whether the gap in conversion is real or just noise.

Problem

“Is the new cart better?”

The team made a new cart page: shorter, with fewer fields. Nobody knows whether it is better. So for two weeks every visit to the site sees, at random, either the old (old) or the new (new) cart. The ordered column is 1 if the visit ends with an order:

Pythondata: the shop tables

import pandas as pd

visits = pd.read_csv('ab_visits.csv')
print(len(visits), visits['day'].min(), visits['day'].max())
print(visits.head(3))
summary = visits.groupby('variant').agg(visits=('ordered', 'size'), orders=('ordered', 'sum'))
summary['rate'] = (summary['orders'] / summary['visits'] * 100).round(2)
summary

import pandas as pd visits = pd.read_csv('ab_visits.csv') print(len(visits), visits['day'].min(), visits['day'].max()) print(visits.head(3)) summary = visits.groupby('variant').agg(visits=('ordered', 'size'), orders=('ordered', 'sum')) summary['rate'] = (summary['orders'] / summary['visits'] * 100).round(2) summary

▶ RunRestore the code

This is an A/B test: two versions, visitors split at random, one measure. The measure here is the conversion: the share of visits that end with an order. The old cart has 3.10%, the new one 3.72%.

The gap is 0.62 percentage points: a difference between two percentages. That is about one fifth more orders. But is it noise?

The idea

Is the gap noise?

This is the question from the lesson on hypotheses. The null hypothesis: the cart makes no difference. If it is true, the labels old and new do not matter and you can shuffle them. How often do the shuffled data give a gap of at least 0.62 points?

Pythondata: the shop tables

import numpy as np

import pandas as pd

visits = pd.read_csv('ab_visits.csv')
ordered = visits['ordered'].to_numpy()
is_new = (visits['variant'] == 'new').to_numpy()
gap = ordered[is_new].mean() - ordered[~is_new].mean()

# Shuffle the labels 2000 times: this is what the gaps look like if the cart makes no difference.
rng = np.random.default_rng(42)
gaps = np.empty(2000)
for i in range(2000):
    shuffled = rng.permutation(is_new)
    gaps[i] = ordered[shuffled].mean() - ordered[~shuffled].mean()
as_big = (np.abs(gaps) >= abs(gap)).sum()
print('the gap:', round(gap * 100, 2), 'points')
print('the largest when shuffled:', round(np.abs(gaps).max() * 100, 2), 'points')
print('as big or bigger:', as_big, 'of 2000, p =', as_big / 2000)

import numpy as np import pandas as pd visits = pd.read_csv('ab_visits.csv') ordered = visits['ordered'].to_numpy() is_new = (visits['variant'] == 'new').to_numpy() gap = ordered[is_new].mean() - ordered[~is_new].mean() # Shuffle the labels 2000 times: this is what the gaps look like if the cart makes no difference. rng = np.random.default_rng(42) gaps = np.empty(2000) for i in range(2000): shuffled = rng.permutation(is_new) gaps[i] = ordered[shuffled].mean() - ordered[~shuffled].mean() as_big = (np.abs(gaps) >= abs(gap)).sum() print('the gap:', round(gap * 100, 2), 'points') print('the largest when shuffled:', round(np.abs(gaps).max() * 100, 2), 'points') print('as big or bigger:', as_big, 'of 2000, p =', as_big / 2000)

▶ RunRestore the code

In 2000 shuffles the largest gap is 0.76 points, but only 6 times is it at least as big as the real one. p is 0.003. A gap like this rarely comes from noise.

More precisely, if you are curious

ordered is 0 or 1, so the mean of the column is exactly the share of orders. The shuffling here is a loop: the table has 28,176 rows, and 2000 shuffles at once in one matrix would take a lot of memory in the browser.

Try it

On every device?

Does the lift come from phones or from desktops? Store in rates the conversion in percent for every device (rows) and variant (columns), and in lift the difference new − old for every device, in points.

Pythondata: the shop tables

import pandas as pd

visits = pd.read_csv('ab_visits.csv')
rates = ...
lift = ...
print(rates)
print(lift)

import pandas as pd visits = pd.read_csv('ab_visits.csv') rates = ... lift = ... print(rates) print(lift)

Check▶ Run

The idea

The quicker way: chi2_contingency

For two shares there is a ready-made check. First pd.crosstab counts the pairs: how many visits of each variant have an order and how many do not. Then scipy.stats.chi2_contingency says how rare such a table is if the variant makes no difference:

Pythondata: the shop tables

import pandas as pd

visits = pd.read_csv('ab_visits.csv')
from scipy import stats

table = pd.crosstab(visits['variant'], visits['ordered'])
print(table)
result = stats.chi2_contingency(table)
print('p =', round(result.pvalue, 4))

import pandas as pd visits = pd.read_csv('ab_visits.csv') from scipy import stats table = pd.crosstab(visits['variant'], visits['ordered']) print(table) result = stats.chi2_contingency(table) print('p =', round(result.pvalue, 4))

▶ RunRestore the code

p is 0.0046, about 0.005. The shuffling gave 0.003: both ways say the same. The small difference is because 2000 shuffles measure p roughly (6 cases), and chi2_contingency makes a small correction for a 2 × 2 table, which lifts p slightly.

The idea

How big is the lift?

p only says “hardly noise”. The manager asks something else: how much better is the new cart? For that you need an interval. As in the lesson on intervals, you draw new samples with replacement from each group and watch how the gap changes:

Pythondata: the shop tables

import numpy as np

import pandas as pd

visits = pd.read_csv('ab_visits.csv')
new = visits.loc[visits['variant'] == 'new', 'ordered'].to_numpy()
old = visits.loc[visits['variant'] == 'old', 'ordered'].to_numpy()

rng = np.random.default_rng(42)
lifts = np.empty(2000)
for i in range(2000):
    lifts[i] = rng.choice(new, len(new)).mean() - rng.choice(old, len(old)).mean()
print('the lift:', round((new.mean() - old.mean()) * 100, 2), 'points')
print('95% interval:', np.percentile(lifts * 100, [2.5, 97.5]).round(2))

import numpy as np import pandas as pd visits = pd.read_csv('ab_visits.csv') new = visits.loc[visits['variant'] == 'new', 'ordered'].to_numpy() old = visits.loc[visits['variant'] == 'old', 'ordered'].to_numpy() rng = np.random.default_rng(42) lifts = np.empty(2000) for i in range(2000): lifts[i] = rng.choice(new, len(new)).mean() - rng.choice(old, len(old)).mean() print('the lift:', round((new.mean() - old.mean()) * 100, 2), 'points') print('95% interval:', np.percentile(lifts * 100, [2.5, 97.5]).round(2))

▶ RunRestore the code

The lift is 0.62 points, and the 95% interval is from 0.21 to 1.04 points. Zero is outside the interval: that is the same “hardly noise”. But the interval also says how much: it may be a little over 0.2, it may be over 1 point.

More precisely, if you are curious

Here there are 2000 repeats, not 10,000, so that it runs fast: each one draws 14,000 rows. By the formula, the interval of a difference between two shares is the difference ± 1.96 × √(p₁(1 − p₁)/n₁ + p₂(1 − p₂)/n₂). It gives almost the same:

Pythondata: the shop tables

import numpy as np

new_rate, new_n = 519 / 13936, 13936
old_rate, old_n = 442 / 14240, 14240
gap = new_rate - old_rate
se = np.sqrt(new_rate * (1 - new_rate) / new_n + old_rate * (1 - old_rate) / old_n)
print(round(gap * 100, 2), round((gap - 1.96 * se) * 100, 2), round((gap + 1.96 * se) * 100, 2))

import numpy as np new_rate, new_n = 519 / 13936, 13936 old_rate, old_n = 442 / 14240, 14240 gap = new_rate - old_rate se = np.sqrt(new_rate * (1 - new_rate) / new_n + old_rate * (1 - old_rate) / old_n) print(round(gap * 100, 2), round((gap - 1.96 * se) * 100, 2), round((gap + 1.96 * se) * 100, 2))

▶ RunRestore the code

The idea

How many visits you need

How long should the test run? You decide that before you start, from two numbers: the conversion so far (3.1%) and the smallest lift that matters to you (0.6 points). There is a rule of thumb for an 80% chance to see the lift at the 0.05 threshold:

visits per group ≈ 16 × p × (1 − p) / lift²

Pythondata: the shop tables

base = 0.031   # the conversion so far
lift = 0.006   # the smallest lift you want to see: 0.6 points
per_group = 16 * base * (1 - base) / lift ** 2
print('per group:', round(per_group))
print('in total:', round(2 * per_group))

base = 0.031 # the conversion so far lift = 0.006 # the smallest lift you want to see: 0.6 points per_group = 16 * base * (1 - base) / lift ** 2 print('per group:', round(per_group)) print('in total:', round(2 * per_group))

▶ RunRestore the code

You need 13,351 visits per group, 26,701 in total. The chance that the test catches a real lift is called power. You can check it with a simulation: make up 2000 tests with conversions of 3.1% and 3.7% and count how often p comes out under 0.05. rng.binomial(n, 0.031) gives how many orders come from n visits, each with a 3.1% chance:

Pythondata: the shop tables

import numpy as np
from scipy import stats

def p_value(orders_a, n_a, orders_b, n_b):
    pooled = (orders_a + orders_b) / (n_a + n_b)
    se = np.sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
    return 2 * stats.norm.sf(np.abs(orders_b / n_b - orders_a / n_a) / se)

rng = np.random.default_rng(42)
for n in [2000, 5000, 13351, 20000]:
    # 2000 made-up tests: the old conversion is 3.1%, the new 3.7%. How often does the test catch it?
    old = rng.binomial(n, 0.031, size=2000)
    new = rng.binomial(n, 0.037, size=2000)
    print(n, 'per group:', (p_value(old, n, new, n) < 0.05).mean())

import numpy as np from scipy import stats def p_value(orders_a, n_a, orders_b, n_b): pooled = (orders_a + orders_b) / (n_a + n_b) se = np.sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b)) return 2 * stats.norm.sf(np.abs(orders_b / n_b - orders_a / n_a) / se) rng = np.random.default_rng(42) for n in [2000, 5000, 13351, 20000]: # 2000 made-up tests: the old conversion is 3.1%, the new 3.7%. How often does the test catch it? old = rng.binomial(n, 0.031, size=2000) new = rng.binomial(n, 0.037, size=2000) print(n, 'per group:', (p_value(old, n, new, n) < 0.05).mean())

▶ RunRestore the code

With 2000 visits per group the test catches the lift in only 18% of cases, with 5000 in 37%. With 13,351 it is 78%, close to the 80% of the rule. A small test does not say “no difference”: it just cannot see.

More precisely, if you are curious

p_value is the check of two shares by formula (a z-test). It works on whole arrays at once, so 2000 tests take a moment. For a 2 × 2 table it gives almost the same as chi2_contingency. The 16 in the rule comes from the 0.05 threshold and 80% power: (1.96 + 0.84)² × 2 ≈ 16.

Try it

A bigger lift

The product team says: “We only care about a lift of at least 1 point.” How many visits per group does the rule need? Store them in per_group as a whole number.

Pythondata: the shop tables

base = 0.031
lift = ...
per_group = ...
print(per_group)

base = 0.031 lift = ... per_group = ... print(per_group)

Check▶ Run

The idea

Do not peek before the end

The temptation is to look at the result every day. Here is p computed from all visits up to each day:

Pythondata: the shop tables

import matplotlib.pyplot as plt

import pandas as pd

visits = pd.read_csv('ab_visits.csv')
from scipy import stats

days = sorted(visits['day'].unique())
p_by_day = []
for day in days:
    so_far = visits[visits['day'] <= day]
    table = pd.crosstab(so_far['variant'], so_far['ordered'])
    p_by_day.append(stats.chi2_contingency(table).pvalue)
p_by_day = pd.Series(p_by_day, index=[d[5:] for d in days])
print(p_by_day.round(3).head(6))

fig, ax = plt.subplots(figsize=(8, 3.5))
ax.plot(p_by_day.index, p_by_day.values, marker='o')
ax.axhline(0.05, color='grey', linestyle='--')
ax.set_title('The p-value day by day')
ax.set_xlabel('day')
ax.set_ylabel('p')
ax.tick_params(axis='x', rotation=45)
fig.tight_layout()

import matplotlib.pyplot as plt import pandas as pd visits = pd.read_csv('ab_visits.csv') from scipy import stats days = sorted(visits['day'].unique()) p_by_day = [] for day in days: so_far = visits[visits['day'] <= day] table = pd.crosstab(so_far['variant'], so_far['ordered']) p_by_day.append(stats.chi2_contingency(table).pvalue) p_by_day = pd.Series(p_by_day, index=[d[5:] for d in days]) print(p_by_day.round(3).head(6)) fig, ax = plt.subplots(figsize=(8, 3.5)) ax.plot(p_by_day.index, p_by_day.values, marker='o') ax.axhline(0.05, color='grey', linestyle='--') ax.set_title('The p-value day by day') ax.set_xlabel('day') ax.set_ylabel('p') ax.tick_params(axis='x', rotation=45) fig.tight_layout()

▶ RunRestore the code

On day three p is 0.053, on day four 0.077, on day five 0.041. Had you stopped on day five, you would have declared a win. Here the new cart really is better, but that shows from the whole test. The rule “I stop as soon as p falls under 0.05” is the mistake itself.

Why? Check it with an A/A test: both groups see the same thing, and there is no difference. We simulate 1000 such tests of 14 days and look at p every day:

Pythondata: the shop tables

import numpy as np
from scipy import stats

def p_value(orders_a, n_a, orders_b, n_b):
    pooled = (orders_a + orders_b) / (n_a + n_b)
    se = np.sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
    return 2 * stats.norm.sf(np.abs(orders_b / n_b - orders_a / n_a) / se)

# 1000 A/A tests: both groups are the same, 3.1% conversion. 14 days of 1000 visits per group.
rng = np.random.default_rng(42)
old = rng.binomial(1000, 0.031, size=(1000, 14)).cumsum(axis=1)
new = rng.binomial(1000, 0.031, size=(1000, 14)).cumsum(axis=1)
seen = np.arange(1, 15) * 1000
p = p_value(old, seen, new, seen)
print('"wins" looking only at the end:', (p[:, -1] < 0.05).mean())
print('"wins" peeking every day:', (p < 0.05).any(axis=1).mean())

import numpy as np from scipy import stats def p_value(orders_a, n_a, orders_b, n_b): pooled = (orders_a + orders_b) / (n_a + n_b) se = np.sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b)) return 2 * stats.norm.sf(np.abs(orders_b / n_b - orders_a / n_a) / se) # 1000 A/A tests: both groups are the same, 3.1% conversion. 14 days of 1000 visits per group. rng = np.random.default_rng(42) old = rng.binomial(1000, 0.031, size=(1000, 14)).cumsum(axis=1) new = rng.binomial(1000, 0.031, size=(1000, 14)).cumsum(axis=1) seen = np.arange(1, 15) * 1000 p = p_value(old, seen, new, seen) print('"wins" looking only at the end:', (p[:, -1] < 0.05).mean()) print('"wins" peeking every day:', (p < 0.05).any(axis=1).mean())

▶ RunRestore the code

If you only look at the end, the false “wins” are 4.9%, as much as the 0.05 threshold promises. If you look every day and stop at the first p under 0.05, they are 21.7%: four times more. The noise has 14 chances to dip under the line.

1. The test is planned for 14 days. On day five p is 0.041. A colleague says: “It is under 0.05, let’s stop and switch on the new cart.” What is wrong?

Nothing: p under 0.05 is proofp should be under 0.01 on day fiveIf you look every day and stop at the first p under 0.05, false “wins” become about 4 times more than 5%

Check

The idea

Many metrics, many segments

The new cart brings more orders. Are they bigger, too? Compare the value per order:

Pythondata: the shop tables

import pandas as pd

visits = pd.read_csv('ab_visits.csv')
from scipy import stats

orders = visits[visits['ordered'] == 1]
print(orders.groupby('variant')['revenue'].mean().round(2))
new = orders.loc[orders['variant'] == 'new', 'revenue']
old = orders.loc[orders['variant'] == 'old', 'revenue']
print('p =', round(stats.ttest_ind(new, old).pvalue, 2))

import pandas as pd visits = pd.read_csv('ab_visits.csv') from scipy import stats orders = visits[visits['ordered'] == 1] print(orders.groupby('variant')['revenue'].mean().round(2)) new = orders.loc[orders['variant'] == 'new', 'revenue'] old = orders.loc[orders['variant'] == 'old', 'revenue'] print('p =', round(stats.ttest_ind(new, old).pvalue, 2))

▶ RunRestore the code

€26.30 with the new cart and €26.71 with the old one, p = 0.73. There is no evidence that the cart changes the order value. That is not “we proved they are equal”: there may be a small difference, but this test cannot see it.

Now picture a report with 20 metrics: orders, value, returns, time on the page… Each metric without a real difference has a 5% chance to come out with p under 0.05. How often does at least one “win”?

Pythondata: the shop tables

import numpy as np
from scipy import stats

def p_value(orders_a, n_a, orders_b, n_b):
    pooled = (orders_a + orders_b) / (n_a + n_b)
    se = np.sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
    return 2 * stats.norm.sf(np.abs(orders_b / n_b - orders_a / n_a) / se)

# 1000 A/A tests with 20 metrics each. Not one metric has a real difference.
rng = np.random.default_rng(42)
old = rng.binomial(14000, 0.031, size=(1000, 20))
new = rng.binomial(14000, 0.031, size=(1000, 20))
p = p_value(old, 14000, new, 14000)
print('metrics with p < 0.05:', (p < 0.05).mean())
print('tests with at least one "win":', (p < 0.05).any(axis=1).mean())

import numpy as np from scipy import stats def p_value(orders_a, n_a, orders_b, n_b): pooled = (orders_a + orders_b) / (n_a + n_b) se = np.sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b)) return 2 * stats.norm.sf(np.abs(orders_b / n_b - orders_a / n_a) / se) # 1000 A/A tests with 20 metrics each. Not one metric has a real difference. rng = np.random.default_rng(42) old = rng.binomial(14000, 0.031, size=(1000, 20)) new = rng.binomial(14000, 0.031, size=(1000, 20)) p = p_value(old, 14000, new, 14000) print('metrics with p < 0.05:', (p < 0.05).mean()) print('tests with at least one "win":', (p < 0.05).any(axis=1).mean())

▶ RunRestore the code

One metric “wins” in 5.1% of cases, as expected. But at least one of 20 “wins” in 64.1% of tests. The same holds for segments: mobile, desktop, new customers, each day of the week. That is why the main metric is chosen before the test. A chance “win” in the 20th metric is an idea for the next test, not a conclusion.

The idea

Why the test says “causes”

Why do you believe the cart causes more orders, and does not just go along with them? Because the variant was assigned at random. Randomness evens out everything you do not measure. Here are the devices in both groups:

Pythondata: the shop tables

import pandas as pd

visits = pd.read_csv('ab_visits.csv')
pd.crosstab(visits['device'], visits['variant'], normalize='columns').round(3)

import pandas as pd visits = pd.read_csv('ab_visits.csv') pd.crosstab(visits['device'], visits['variant'], normalize='columns').round(3)

▶ RunRestore the code

In both groups desktops are 35% and phones 65%. The same goes for the hour, the day, people’s habits: the groups differ only in the cart. So the gap in orders comes from it.

When the data are not from a test, this is not so. In the next lesson a banner seems to lift orders, yet on each device on its own it does not.

Exercise

A report for the manager

The manager asks: “Does the new cart bring more money per visit?” Store in per_visit the average revenue per visit for each variant (all visits, including those without an order), and in p the p-value of the comparison with scipy.stats.ttest_ind.

Pythondata: the shop tables

import pandas as pd

visits = pd.read_csv('ab_visits.csv')
from scipy import stats

per_visit = ...
p = ...
print(per_visit)
print(p)

import pandas as pd visits = pd.read_csv('ab_visits.csv') from scipy import stats per_visit = ... p = ... print(per_visit) print(p)

Check▶ Run

Had you planned it from the start: how many days should the test have run? The visits per day are len(visits) / 14 on average. Store in per_group the visits per group by the rule (a lift of 0.6 points at 3.1%), in per_day the visits per day, and in days the days, rounded up.

Pythondata: the shop tables

import math
import pandas as pd

visits = pd.read_csv('ab_visits.csv')
base = 0.031
lift = 0.006
per_group = ...
per_day = ...
days = ...
print(days)

import math import pandas as pd visits = pd.read_csv('ab_visits.csv') base = 0.031 lift = 0.006 per_group = ... per_day = ... days = ... print(days)

Check▶ Run

Test

Check yourself

1. The check of the new cart gives p = 0.005. What does that mean?

If the cart made no difference, a gap at least this big would come up by chance in about 5 of 1000 testsThe lift is bigThe new cart is better with a probability of 99.5%

2. You want to see a lift of 0.3 points instead of 0.6. How many visits do you need?

About 4 times moreAbout 2 times moreAs many as before

3. The report tracks 20 metrics. One of them has p = 0.04, the others do not. What do you say?

The new version improves that metricThe test is brokenIt is probably chance: with 20 metrics and no difference at all, at least one “wins” in about 64% of tests

4. The average order is €26.30 with the new cart and €26.71 with the old one, p = 0.73. Which conclusion is right?

The new cart lowers the order value by €0.41We proved that the order value is the sameThere is no evidence that the cart changes the order value

5. Why can you say “the new cart leads to more orders” after the A/B test, and not just “the two go together”?

Because p is smallThe variant was assigned at random, so the two groups are alike in everything elseBecause there are many visits

Check

Start the lesson over

Related lessons

• Builds on: The first chart · Sample and population · Confidence intervals · Testing a hypothesis
• Continues in: Correlation and cause
• Tasks on this topic: Statistics for decisions