Confidence intervals

In short: A confidence interval is a range around the value from a sample that, at 95%, catches the true value in about 95 of 100 samples; the bootstrap builds it from resamples with replacement.

Problem

“How much does a customer spend per order?”

The manager asks what the average order is. In real life you rarely have everything: say you have 200 orders, drawn at random from the year. Here we also have the whole year, 10,459 orders without the cancelled ones, and we can see what happens:

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
cancelled = orders.loc[orders['status'] == 'cancelled', 'id']
items = items[~items['order_id'].isin(cancelled)]
items['revenue'] = items['quantity'] * items['unit_price']
totals = items.groupby('order_id')['revenue'].sum()

print(len(totals))
rng = np.random.default_rng(1)
for i in range(3):
    sample = rng.choice(totals.to_numpy(), size=200, replace=False)
    print(round(sample.mean(), 2))

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') cancelled = orders.loc[orders['status'] == 'cancelled', 'id'] items = items[~items['order_id'].isin(cancelled)] items['revenue'] = items['quantity'] * items['unit_price'] totals = items.groupby('order_id')['revenue'].sum() print(len(totals)) rng = np.random.default_rng(1) for i in range(3): sample = rng.choice(totals.to_numpy(), size=200, replace=False) print(round(sample.mean(), 2))

▶ RunRestore the code

Three samples of 200 give three different means: €9.77, €10.74 and €11.16. You know from the lesson on samples that the mean is noisy. But you only have one sample. What do you tell the manager: one number or something more?

It is more honest to say “the average is somewhere between … and …”. What is left is to find out how wide that “somewhere” should be, from the sample you have and nothing else.

The idea

The sample instead of the year: the bootstrap

If you could draw new samples from the year, you would see how much the mean swings. You cannot. So you use the sample as a small model of the year: you draw 200 orders from it with replacement and compute the mean. This is a resample. One order comes up twice, another not at all, and the mean is a little different each time.

You do it 10,000 times and look at where the means are. You cut off 2.5% of the smallest and 2.5% of the largest. The middle 95% is the confidence interval. The method is called the bootstrap:

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
cancelled = orders.loc[orders['status'] == 'cancelled', 'id']
items = items[~items['order_id'].isin(cancelled)]
items['revenue'] = items['quantity'] * items['unit_price']
totals = items.groupby('order_id')['revenue'].sum()

rng = np.random.default_rng(42)
sample = rng.choice(totals.to_numpy(), size=200, replace=False)
print(sample[:8])
boots = rng.choice(sample, size=(10_000, 200)).mean(axis=1)
print(boots[:3].round(2))
low, high = np.quantile(boots, [0.025, 0.975])
print(round(sample.mean(), 2), round(low, 2), round(high, 2))
print(round(totals.mean(), 2))

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') cancelled = orders.loc[orders['status'] == 'cancelled', 'id'] items = items[~items['order_id'].isin(cancelled)] items['revenue'] = items['quantity'] * items['unit_price'] totals = items.groupby('order_id')['revenue'].sum() rng = np.random.default_rng(42) sample = rng.choice(totals.to_numpy(), size=200, replace=False) print(sample[:8]) boots = rng.choice(sample, size=(10_000, 200)).mean(axis=1) print(boots[:3].round(2)) low, high = np.quantile(boots, [0.025, 0.975]) print(round(sample.mean(), 2), round(low, 2), round(high, 2)) print(round(totals.mean(), 2))

▶ RunRestore the code

The first 8 orders of the sample are from €0.90 to €45.20. The first three resamples give means of €10.07, €11.07 and €11.13. The mean of the sample is €11.41, and the 95% interval is from €10.04 to €12.83. The true mean of the year is €10.63: it is inside.

rng.choice(sample, size=(10_000, 200)) is a 10,000 × 200 table: every row is one resample. mean(axis=1) gives one mean per row. No loop needed.

See it

How an interval is born from one sample

The same code, slowly: tiles from the sample are copied into a resample, its mean drops onto the axis, and 10,000 such means make a pile.

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
cancelled = orders.loc[orders['status'] == 'cancelled', 'id']
items = items[~items['order_id'].isin(cancelled)]
items['revenue'] = items['quantity'] * items['unit_price']
totals = items.groupby('order_id')['revenue'].sum()

rng = np.random.default_rng(42)
sample = rng.choice(totals.to_numpy(), size=200, replace=False)
print(sample[:8])
boots = rng.choice(sample, size=(10_000, 200)).mean(axis=1)
print(boots[:3].round(2))
low, high = np.quantile(boots, [0.025, 0.975])
print(round(sample.mean(), 2), round(low, 2), round(high, 2))
print(round(totals.mean(), 2))

Step by step, in words

1. totals is the value of every order of the year. In real life you do not see the whole: you only have a sample.
2. sample is a sample of 200 orders. At the top are the first 8, in euros: 3.6, 0.9, 2.1, 17, 45.2, 13.2, 3.4 and 2.
3. A resample: 200 draws from sample with replacement. One order comes up twice, another not at all.
4. The mean of the resample drops as a dot onto the axis. The first three are €10.07, €11.07 and €11.13.
5. Like this 10,000 times. The dots pile up around the mean of the sample, €11.41.
6. Cut off 2.5% on the left and 2.5% on the right. The middle 95% is the interval: from €10.04 to €12.83.
7. The true mean of the year is €10.63, and it is inside. In real life you do not know it: the interval says how far off it may be.

The pile in the scene is a sketch: the numbers in the captions are real, from the sample to the limits of the interval. Here are the real 10,000 means with matplotlib, with the two limits (blue lines) and the true mean (the orange dashed one):

Pythondata: the shop tables

import matplotlib.pyplot as plt

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
cancelled = orders.loc[orders['status'] == 'cancelled', 'id']
items = items[~items['order_id'].isin(cancelled)]
items['revenue'] = items['quantity'] * items['unit_price']
totals = items.groupby('order_id')['revenue'].sum()

rng = np.random.default_rng(42)
sample = rng.choice(totals.to_numpy(), size=200, replace=False)
boots = rng.choice(sample, size=(10_000, 200)).mean(axis=1)
low, high = np.quantile(boots, [0.025, 0.975])

fig, ax = plt.subplots(figsize=(7, 3.5))
ax.hist(boots, bins=40, color='#9aa3ad')
ax.axvline(low, color='#0072B2')
ax.axvline(high, color='#0072B2')
ax.axvline(totals.mean(), color='#D55E00', linestyle='--')
ax.set_title('10,000 means of resamples')
ax.set_xlabel('mean order, €')
ax.set_ylabel('count')

import matplotlib.pyplot as plt import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') cancelled = orders.loc[orders['status'] == 'cancelled', 'id'] items = items[~items['order_id'].isin(cancelled)] items['revenue'] = items['quantity'] * items['unit_price'] totals = items.groupby('order_id')['revenue'].sum() rng = np.random.default_rng(42) sample = rng.choice(totals.to_numpy(), size=200, replace=False) boots = rng.choice(sample, size=(10_000, 200)).mean(axis=1) low, high = np.quantile(boots, [0.025, 0.975]) fig, ax = plt.subplots(figsize=(7, 3.5)) ax.hist(boots, bins=40, color='#9aa3ad') ax.axvline(low, color='#0072B2') ax.axvline(high, color='#0072B2') ax.axvline(totals.mean(), color='#D55E00', linestyle='--') ax.set_title('10,000 means of resamples') ax.set_xlabel('mean order, €') ax.set_ylabel('count')

▶ RunRestore the code

Try it

Without replacement?

1. In the bootstrap every resample draws 200 orders from a sample of 200. What happens if you draw without replacement?

Nothing: with and without replacement give the sameThe interval becomes more preciseEvery resample is the same sample, only shuffled: the mean is always €11.41 and the interval is zero

Check

The median of the orders is more robust to the big orders. Make a 95% interval for the median with the same sample of 200: put the medians of 10,000 resamples in meds, and the limits of the interval in low and high.

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
cancelled = orders.loc[orders['status'] == 'cancelled', 'id']
items = items[~items['order_id'].isin(cancelled)]
items['revenue'] = items['quantity'] * items['unit_price']
totals = items.groupby('order_id')['revenue'].sum()

rng = np.random.default_rng(42)
sample = rng.choice(totals.to_numpy(), size=200, replace=False)
meds = ...
low, high = ...
print(np.median(sample), low, high)

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') cancelled = orders.loc[orders['status'] == 'cancelled', 'id'] items = items[~items['order_id'].isin(cancelled)] items['revenue'] = items['quantity'] * items['unit_price'] totals = items.groupby('order_id')['revenue'].sum() rng = np.random.default_rng(42) sample = rng.choice(totals.to_numpy(), size=200, replace=False) meds = ... low, high = ... print(np.median(sample), low, high)

Check▶ Run

The idea

What 95% means

The true mean of the year is one number: €10.63. The interval from one sample either catches it or not. 95% is a promise of the method: if you draw many samples and build an interval for each, about 95 of 100 intervals will catch the truth. This is called the confidence level.

Here we have the whole year and can check the promise. 100 samples of 200, for each a bootstrap with 1,000 resamples:

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
cancelled = orders.loc[orders['status'] == 'cancelled', 'id']
items = items[~items['order_id'].isin(cancelled)]
items['revenue'] = items['quantity'] * items['unit_price']
totals = items.groupby('order_id')['revenue'].sum()

truth = totals.mean()
rng = np.random.default_rng(7)
caught = 0
for i in range(100):
    sample = rng.choice(totals.to_numpy(), size=200, replace=False)
    boots = rng.choice(sample, size=(1_000, 200)).mean(axis=1)
    low, high = np.quantile(boots, [0.025, 0.975])
    if low <= truth <= high:
        caught += 1
print(caught)

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') cancelled = orders.loc[orders['status'] == 'cancelled', 'id'] items = items[~items['order_id'].isin(cancelled)] items['revenue'] = items['quantity'] * items['unit_price'] totals = items.groupby('order_id')['revenue'].sum() truth = totals.mean() rng = np.random.default_rng(7) caught = 0 for i in range(100): sample = rng.choice(totals.to_numpy(), size=200, replace=False) boots = rng.choice(sample, size=(1_000, 200)).mean(axis=1) low, high = np.quantile(boots, [0.025, 0.975]) if low <= truth <= high: caught += 1 print(caught)

▶ RunRestore the code

93 of 100 intervals catch €10.63. That is close to 95: the 100 samples are random themselves, and the next 100 will give something a little different. About 5 of 100 honest intervals miss the truth, and there is no way to know which ones.

More precisely, if you are curious

It is tempting to say “with probability 95% the true mean is between €10.04 and €12.83”. Strictly speaking, that is not it: the true mean is not random, and the interval from your sample either contains it or not. What is random is the sample. So 95% is about the method, not about one particular number. In a report write “95% confidence interval: €10.04–€12.83”.

The bootstrap works when the sample is random. If the 200 orders are from one December, the interval will be for December, however many resamples you make: resamples do not fix a skewed sample.

1. The interval for the average order is from €10.04 to €12.83. Which sentence is true?

95% of the orders are between €10.04 and €12.83The average order is €11.41, with an error of at most €1.42The method that gives it catches the true mean in about 95 of 100 samples

Check

The idea

The formula as a shortcut

For a mean there is a shortcut without 10,000 resamples. The noise of the mean is the standard error: std / √n. With a normal distribution the middle 95% lies within 1.96 standard errors of the centre. So the interval is the mean ± 1.96 × std / √n:

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
cancelled = orders.loc[orders['status'] == 'cancelled', 'id']
items = items[~items['order_id'].isin(cancelled)]
items['revenue'] = items['quantity'] * items['unit_price']
totals = items.groupby('order_id')['revenue'].sum()

rng = np.random.default_rng(42)
sample = rng.choice(totals.to_numpy(), size=200, replace=False)
se = sample.std(ddof=1) / np.sqrt(len(sample))
low = sample.mean() - 1.96 * se
high = sample.mean() + 1.96 * se
print(round(se, 2), round(low, 2), round(high, 2))

samples = rng.choice(totals.to_numpy(), size=(1_000, 200))
means = samples.mean(axis=1)
ses = samples.std(axis=1, ddof=1) / np.sqrt(200)
inside = (means - 1.96 * ses <= totals.mean()) & (totals.mean() <= means + 1.96 * ses)
print(inside.sum())

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') cancelled = orders.loc[orders['status'] == 'cancelled', 'id'] items = items[~items['order_id'].isin(cancelled)] items['revenue'] = items['quantity'] * items['unit_price'] totals = items.groupby('order_id')['revenue'].sum() rng = np.random.default_rng(42) sample = rng.choice(totals.to_numpy(), size=200, replace=False) se = sample.std(ddof=1) / np.sqrt(len(sample)) low = sample.mean() - 1.96 * se high = sample.mean() + 1.96 * se print(round(se, 2), round(low, 2), round(high, 2)) samples = rng.choice(totals.to_numpy(), size=(1_000, 200)) means = samples.mean(axis=1) ses = samples.std(axis=1, ddof=1) / np.sqrt(200) inside = (means - 1.96 * ses <= totals.mean()) & (totals.mean() <= means + 1.96 * ses) print(inside.sum())

▶ RunRestore the code

The standard error is €0.72, and the interval is from €10.00 to €12.81. The bootstrap gave €10.04–€12.83: almost the same. The second line checks the promise of the formula: out of 1,000 samples of 200, 948 intervals catch the truth. Close to 950.

std(ddof=1) divides by n − 1 instead of n, like std() in pandas. With 200 orders the difference is in the third digit.

More precisely, if you are curious

The number 1.96 comes from the normal distribution: 95% of the area lies between −1.96 and 1.96 standard deviations. The mean of 200 orders is close to normal even when the orders themselves have a long tail (the central limit theorem from the lesson on samples). With small samples, under 30, or with a very long tail the formula is further off, and the bootstrap is more honest.

The idea

An interval for a share

What part of the orders is cancelled? A share is a mean of zeros and ones: True is 1, False is 0, and mean() gives the share. So the bootstrap and the formula work here too. The formula for a share is p ± 1.96 × √(p(1 − p) / n):

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
is_cancelled = (orders['status'] == 'cancelled').to_numpy()

rng = np.random.default_rng(5)
sample = rng.choice(is_cancelled, size=500, replace=False)
p = sample.mean()
boots = rng.choice(sample, size=(10_000, 500)).mean(axis=1)
print(p, np.quantile(boots, [0.025, 0.975]))
se = np.sqrt(p * (1 - p) / 500)
print(round(p - 1.96 * se, 3), round(p + 1.96 * se, 3))
print(round(is_cancelled.mean(), 3))

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') is_cancelled = (orders['status'] == 'cancelled').to_numpy() rng = np.random.default_rng(5) sample = rng.choice(is_cancelled, size=500, replace=False) p = sample.mean() boots = rng.choice(sample, size=(10_000, 500)).mean(axis=1) print(p, np.quantile(boots, [0.025, 0.975])) se = np.sqrt(p * (1 - p) / 500) print(round(p - 1.96 * se, 3), round(p + 1.96 * se, 3)) print(round(is_cancelled.mean(), 3))

▶ RunRestore the code

In a sample of 500 orders 5.8% are cancelled. The bootstrap gives from 3.8 to 8.0%, and the formula from 3.8 to 7.8%. The true share for the year is 5.9%. From 500 orders you know the share to about ± 2 points: “5.8%” without an interval sounds more precise than it is.

The idea

A small sample, a wide interval

The fewer the orders, the noisier the mean and the wider the interval:

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
cancelled = orders.loc[orders['status'] == 'cancelled', 'id']
items = items[~items['order_id'].isin(cancelled)]
items['revenue'] = items['quantity'] * items['unit_price']
totals = items.groupby('order_id')['revenue'].sum()

rng = np.random.default_rng(3)
for n in [50, 200, 800]:
    sample = rng.choice(totals.to_numpy(), size=n, replace=False)
    boots = rng.choice(sample, size=(10_000, n)).mean(axis=1)
    low, high = np.quantile(boots, [0.025, 0.975])
    print(n, round(low, 2), round(high, 2), round(high - low, 2))

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') cancelled = orders.loc[orders['status'] == 'cancelled', 'id'] items = items[~items['order_id'].isin(cancelled)] items['revenue'] = items['quantity'] * items['unit_price'] totals = items.groupby('order_id')['revenue'].sum() rng = np.random.default_rng(3) for n in [50, 200, 800]: sample = rng.choice(totals.to_numpy(), size=n, replace=False) boots = rng.choice(sample, size=(10_000, n)).mean(axis=1) low, high = np.quantile(boots, [0.025, 0.975]) print(n, round(low, 2), round(high, 2), round(high - low, 2))

▶ RunRestore the code

50 orders give an interval €4.73 wide, 200 orders €2.69, and 800 orders €1.35. 4 times more data give an interval about half as wide, because the standard error falls with the square root of n. Precision is expensive: halving the width again needs 4 times more orders again.

Exercise

The answer for the manager

A new sample of 400 orders (seed 2026). Make a 95% interval for the average order in two ways: with a bootstrap of 10,000 resamples (boot_low, boot_high) and with the formula (se is the standard error, f_low and f_high are the limits).

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
cancelled = orders.loc[orders['status'] == 'cancelled', 'id']
items = items[~items['order_id'].isin(cancelled)]
items['revenue'] = items['quantity'] * items['unit_price']
totals = items.groupby('order_id')['revenue'].sum()

rng = np.random.default_rng(2026)
sample = rng.choice(totals.to_numpy(), size=400, replace=False)
boots = ...
boot_low, boot_high = ...
se = ...
f_low = ...
f_high = ...

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') cancelled = orders.loc[orders['status'] == 'cancelled', 'id'] items = items[~items['order_id'].isin(cancelled)] items['revenue'] = items['quantity'] * items['unit_price'] totals = items.groupby('order_id')['revenue'].sum() rng = np.random.default_rng(2026) sample = rng.choice(totals.to_numpy(), size=400, replace=False) boots = ... boot_low, boot_high = ... se = ... f_low = ... f_high = ...

Check▶ Run

The manager is thinking about free delivery over €20. What part of the orders is over €20? From a sample of 300 orders (seed 7) put the share in p, and its 95% interval in low and high.

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
cancelled = orders.loc[orders['status'] == 'cancelled', 'id']
items = items[~items['order_id'].isin(cancelled)]
items['revenue'] = items['quantity'] * items['unit_price']
totals = items.groupby('order_id')['revenue'].sum()

rng = np.random.default_rng(7)
sample = rng.choice(totals.to_numpy(), size=300, replace=False)
big = ...
p = ...
low, high = ...
print(round(p, 3), round(low, 3), round(high, 3))

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') cancelled = orders.loc[orders['status'] == 'cancelled', 'id'] items = items[~items['order_id'].isin(cancelled)] items['revenue'] = items['quantity'] * items['unit_price'] totals = items.groupby('order_id')['revenue'].sum() rng = np.random.default_rng(7) sample = rng.choice(totals.to_numpy(), size=300, replace=False) big = ... p = ... low, high = ... print(round(p, 3), round(low, 3), round(high, 3))

Check▶ Run

Test

Test yourself

1. Why does the bootstrap take the 0.025 and 0.975 quantiles of the means, not 0.05 and 0.95?

Because 0.05 is the threshold of the p-valueYou cut off 2.5% on each side: the middle 95% remainsSo that the interval is narrower

2. The interval from 200 orders is €2.69 wide. How many orders do you need for an interval about €1.35 wide?

About 100: the smaller sample is more preciseAbout 400: twice as manyAbout 800: 4 times more orders give an interval half as wide

3. The formula is the mean ± 1.96 × std / √n. When do you prefer the bootstrap?

When you want an interval for a median or another calculation that has no formula at handNever: the bootstrap is only for learningAlways: the formula is wrong

4. In a sample of 500 orders 5.8% are cancelled, and the interval is from 3.8 to 7.8%. The boss says: “So cancellations are 5.8%.” What do you answer?

We know nothing: the sample is smallAbout 5.8%, but from 500 orders we only know that they are somewhere between 3.8 and 7.8%Exactly 5.8%: it is calculated

Check

Start the lesson over

Related lessons

• Builds on: The first chart · Distributions · Describing the data · Sample and population
• Continues in: Testing a hypothesis · A/B test · Why did revenue fall?
• Tasks on this topic: Statistics for decisions