Sample and population

In short: A sample is a part of the population, and its mean jumps from sample to sample; the noise shrinks with the square root of the size: 4 times more data gives half the noise.

Problem

Two samples, two means

The manager wants to know the mean order. He does not want to wait for the year report: he takes 30 orders from the system and gets €12.11. His colleague takes another 30 and gets €10.33.

Who is right? Neither of them made a mistake in the calculation. Only the orders they took are different. How much can you trust a mean of 30 orders? And of 3000?

The idea

Population and sample

Everything you are asking about is called the population: here, all orders of the year without the cancelled ones. The part you looked at is a sample.

Usually you do not see the whole population: all customers, all future orders. Here you have it, so you can check how wrong a sample is. The truth is:

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())
# Cancelled orders are not revenue: leave them out.
values = orders[orders['status'] != 'cancelled']['value'].to_numpy()
true_mean = values.mean()
print('orders:', len(values))
print('mean:', round(true_mean, 2))
print('standard deviation:', round(values.std(), 2))

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) # Cancelled orders are not revenue: leave them out. values = orders[orders['status'] != 'cancelled']['value'].to_numpy() true_mean = values.mean() print('orders:', len(values)) print('mean:', round(true_mean, 2)) print('standard deviation:', round(values.std(), 2))

▶ RunRestore the code

The mean of all 10,459 orders is €10.63. Every sample will be compared with this number.

The idea

A random sample with NumPy

np.random.default_rng(42) creates a random number generator. 42 is the seed: the starting point. With the same seed every run gives the same numbers. So the examples here always show the same, and you can check every result.

rng.choice(values, size=30, replace=False) picks 30 different orders at random. Three samples of each size:

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())
# Cancelled orders are not revenue: leave them out.
values = orders[orders['status'] != 'cancelled']['value'].to_numpy()
true_mean = values.mean()
rng = np.random.default_rng(42)
for n in [30, 300, 3000]:
    means = [rng.choice(values, size=n, replace=False).mean() for _ in range(3)]
    print(n, 'orders:', np.round(means, 2))

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) # Cancelled orders are not revenue: leave them out. values = orders[orders['status'] != 'cancelled']['value'].to_numpy() true_mean = values.mean() rng = np.random.default_rng(42) for n in [30, 300, 3000]: means = [rng.choice(values, size=n, replace=False).mean() for _ in range(3)] print(n, 'orders:', np.round(means, 2))

▶ RunRestore the code

The means of 30 orders are €11.33, €12.11 and €10.33: the manager's and the colleague's numbers. With 300 orders they range from €10.07 to €11.42. With 3000 they range from €10.65 to €11.02: closer to each other and to the truth of €10.63.

1. You run the example a second time without changing anything. Which means do you see?

The same: default_rng(42) starts from the same place every timeThe same only for 3000 ordersDifferent ones, because the choice is random

Check

The idea

A thousand samples

Three samples are too few to see how much the mean jumps. Let us take 1000 of each size. rng.choice(values, size=(1000, n)) gives a 1000 × n table: every row is one sample. mean(axis=1) gives the mean of every row: 1000 means.

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())
# Cancelled orders are not revenue: leave them out.
values = orders[orders['status'] != 'cancelled']['value'].to_numpy()
true_mean = values.mean()
import matplotlib.pyplot as plt

rng = np.random.default_rng(42)
fig, axes = plt.subplots(1, 3, figsize=(12, 3.5), sharex=True)
for ax, n in zip(axes, [30, 300, 3000]):
    # 1000 samples of n orders: one per row.
    means = rng.choice(values, size=(1000, n)).mean(axis=1)
    low, high = np.quantile(means, [0.05, 0.95])
    print(n, 'orders: 90% of the means are between', round(low, 2), 'and', round(high, 2))
    ax.hist(means, bins=30)
    ax.axvline(true_mean, color='black')
    ax.set_title(f'{n} orders')
    ax.set_xlabel('sample mean, €')
fig.suptitle('1000 samples of each size')
fig.tight_layout()

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) # Cancelled orders are not revenue: leave them out. values = orders[orders['status'] != 'cancelled']['value'].to_numpy() true_mean = values.mean() import matplotlib.pyplot as plt rng = np.random.default_rng(42) fig, axes = plt.subplots(1, 3, figsize=(12, 3.5), sharex=True) for ax, n in zip(axes, [30, 300, 3000]): # 1000 samples of n orders: one per row. means = rng.choice(values, size=(1000, n)).mean(axis=1) low, high = np.quantile(means, [0.05, 0.95]) print(n, 'orders: 90% of the means are between', round(low, 2), 'and', round(high, 2)) ax.hist(means, bins=30) ax.axvline(true_mean, color='black') ax.set_title(f'{n} orders') ax.set_xlabel('sample mean, €') fig.suptitle('1000 samples of each size') fig.tight_layout()

▶ RunRestore the code

Every histogram is 1000 means, and the black line is the truth. With 30 orders, 90% of the means are between €7.99 and €13.85: a sample of 30 can show almost anything. With 300 they are between €9.80 and €11.50, and with 3000 between €10.35 and €10.91.

sharex=True gives the three charts the same horizontal axis, so they can be compared. Without it each chart would stretch to its own means and all three would look equally wide.

More precisely, if you are curious

Here the samples are with replacement (replace=True by default): one order may come up twice in one sample. That allows 1000 samples at once, in one table. With 30 or 3000 out of 10,459 orders the difference from a sample without replacement is small.

The orders have a long tail to the right, yet the histograms of the means are symmetric, like a bell. The mean of enough orders behaves like this almost whatever the orders themselves look like: the longer the tail, the more orders the bell needs. This is called the central limit theorem.

The idea

The law of large numbers

What happens if you pile up orders one by one and compute the mean so far after each? np.cumsum gives the sum up to every position. Divide it by the count up to there (np.arange(1, 3001)) and you have the mean after 1, 2, … 3000 orders:

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())
# Cancelled orders are not revenue: leave them out.
values = orders[orders['status'] != 'cancelled']['value'].to_numpy()
true_mean = values.mean()
import matplotlib.pyplot as plt

rng = np.random.default_rng(7)
sample = rng.choice(values, size=3000, replace=False)
running = np.cumsum(sample) / np.arange(1, 3001)
for n in [10, 100, 1000, 3000]:
    print('after', n, 'orders:', round(running[n - 1], 2))

fig, ax = plt.subplots()
ax.plot(np.arange(1, 3001), running)
ax.axhline(true_mean, color='black', linestyle='--')
ax.set_xscale('log')
ax.set_title('The mean settles as the sample grows')
ax.set_xlabel('orders in the sample')
ax.set_ylabel('mean so far, €')

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) # Cancelled orders are not revenue: leave them out. values = orders[orders['status'] != 'cancelled']['value'].to_numpy() true_mean = values.mean() import matplotlib.pyplot as plt rng = np.random.default_rng(7) sample = rng.choice(values, size=3000, replace=False) running = np.cumsum(sample) / np.arange(1, 3001) for n in [10, 100, 1000, 3000]: print('after', n, 'orders:', round(running[n - 1], 2)) fig, ax = plt.subplots() ax.plot(np.arange(1, 3001), running) ax.axhline(true_mean, color='black', linestyle='--') ax.set_xscale('log') ax.set_title('The mean settles as the sample grows') ax.set_xlabel('orders in the sample') ax.set_ylabel('mean so far, €')

▶ RunRestore the code

After 10 orders the mean is €11.29, after 100 it is €9.82, after 1000 it is €10.96, and after 3000 it is €10.65. At first it jumps a lot, then it settles around the truth. This is the law of large numbers: the bigger a random sample, the closer its mean is to the mean of the population.

set_xscale('log') puts 1, 10, 100, 1000 at equal distances along the horizontal axis. That way both the start, where everything happens, and the end are visible.

The idea

The noise shrinks with the square root of n

How much more precise does the mean get when you take more orders? Compare 100, 400 and 1600: each size is 4 times the one before.

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())
# Cancelled orders are not revenue: leave them out.
values = orders[orders['status'] != 'cancelled']['value'].to_numpy()
true_mean = values.mean()
rng = np.random.default_rng(42)
for n in [100, 400, 1600]:
    means = rng.choice(values, size=(1000, n)).mean(axis=1)
    print(n, 'orders: noise', round(means.std(), 3), ' formula', round(values.std() / np.sqrt(n), 3))

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) # Cancelled orders are not revenue: leave them out. values = orders[orders['status'] != 'cancelled']['value'].to_numpy() true_mean = values.mean() rng = np.random.default_rng(42) for n in [100, 400, 1600]: means = rng.choice(values, size=(1000, n)).mean(axis=1) print(n, 'orders: noise', round(means.std(), 3), ' formula', round(values.std() / np.sqrt(n), 3))

▶ RunRestore the code

The noise is the std of the 1000 means: €0.93 with 100 orders, €0.447 with 400 and €0.238 with 1600. Each time, 4 times more orders give about half the noise. The noise shrinks with the square root of the size, not with the size itself.

So there is a short path without a simulation: the standard deviation of the orders divided by the square root of n. This is called the standard error: 9.43 / √100 ≈ €0.94. The formula and the simulation give almost the same (€0.472 against €0.447 with 400; the gap is the noise of the simulation itself).

The price is high: a mean twice as precise needs 4 times more data, and ten times as precise needs 100 times more.

Try it

The noise of 400 orders

How big is the noise of the mean of 400 orders? Draw 1000 samples of 400 orders and store the standard deviation of their means in noise.

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())
# Cancelled orders are not revenue: leave them out.
values = orders[orders['status'] != 'cancelled']['value'].to_numpy()
true_mean = values.mean()
rng = np.random.default_rng(1)
means = ...
noise = ...
print(noise)

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) # Cancelled orders are not revenue: leave them out. values = orders[orders['status'] != 'cancelled']['value'].to_numpy() true_mean = values.mean() rng = np.random.default_rng(1) means = ... noise = ... print(noise)

Check▶ Run

The idea

“December fell tenfold”

In the first lesson of the course the small shop's revenue in December was €22.70, and in October €255. The table has 53 orders without the cancelled ones, over 14 months: about 4 a month. December has 2 of them, October 7.

If all months were the same, what gap would chance alone give? We send each of the 53 orders to a random month and compute the revenue of every month. 1000 times:

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('orders.csv')
items = pd.read_csv('order_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())
values = orders[orders['status'] != 'cancelled']['value'].to_numpy()
print('orders:', len(values))

# 1000 made-up years: every order goes to a random one of the 14 months.
rng = np.random.default_rng(42)
month = rng.integers(0, 14, size=(1000, len(values)))
revenue = np.zeros((1000, 14))
for m in range(14):
    revenue[:, m] = np.where(month == m, values, 0).sum(axis=1)
ratio = revenue.max(axis=1) / revenue.min(axis=1)
print('the strongest month is at least 10 times the weakest:', round((ratio >= 10).mean() * 100, 1), '%')

import numpy as np import pandas as pd orders = pd.read_csv('orders.csv') items = pd.read_csv('order_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) values = orders[orders['status'] != 'cancelled']['value'].to_numpy() print('orders:', len(values)) # 1000 made-up years: every order goes to a random one of the 14 months. rng = np.random.default_rng(42) month = rng.integers(0, 14, size=(1000, len(values))) revenue = np.zeros((1000, 14)) for m in range(14): revenue[:, m] = np.where(month == m, values, 0).sum(axis=1) ratio = revenue.max(axis=1) / revenue.min(axis=1) print('the strongest month is at least 10 times the weakest:', round((ratio >= 10).mean() * 100, 1), '%')

▶ RunRestore the code

rng.integers(0, 14, ...) gives every order a month number from 0 to 13. np.where(month == m, values, 0) keeps the value only for the orders of month m, and sum(axis=1) adds them up for every made-up year.

In 85.6% of the made-up years the strongest month is at least 10 times the weakest. And in them the months are the same by design. The tenfold drop is a sign of nothing: with 4 orders a month it is the usual noise.

In the year data a month has about 870 orders on average, and the noise is much smaller. That is why December there really is the strongest. The September drop is a different question: whether it is noise, and why it happened, we will find out in the project.

Exercise

How many orders are enough?

The manager wants the mean order within ±€0.50 in 95 cases out of 100. For samples of 100, 300, 1000, 2000 and 3000 orders, draw 1000 samples each and store in close (a dict: size → share) how many of the means are at most €0.50 from the true one. Store the smallest size that is enough in enough.

Pythondata: the shop tables

import numpy as np
import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())
# Cancelled orders are not revenue: leave them out.
values = orders[orders['status'] != 'cancelled']['value'].to_numpy()
true_mean = values.mean()
rng = np.random.default_rng(3)
close = {}
for n in [100, 300, 1000, 2000, 3000]:
    means = ...
    close[n] = ...
print(close)
enough = ...
print(enough)

import numpy as np import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) # Cancelled orders are not revenue: leave them out. values = orders[orders['status'] != 'cancelled']['value'].to_numpy() true_mean = values.mean() rng = np.random.default_rng(3) close = {} for n in [100, 300, 1000, 2000, 3000]: means = ... close[n] = ... print(close) enough = ... print(enough)

Check▶ Run

Test

Check yourself

1. The mean of 100 orders has a noise of about €0.94. What is the noise with 400 orders?

Still about €0.94About €0.47: 4 times more data gives half the noiseAbout €0.24: 4 times less

2. The standard deviation of the orders is €9.43, and the standard error of the mean of 100 orders is €0.94. What is the difference?

€0.94 is a mistake in the calculation€9.43 is how much single orders differ; €0.94 is how much the mean moves from sample to sampleThey are the same, only rounded differently

3. A sample of 300 orders from the year gives a mean with a noise of about €0.54. If the shop had a million orders instead of 10,459, what would the noise of 300 of them be?

A hundred times smallerAlmost the same: the noise depends on the size of the sample, not on the size of the populationA hundred times bigger

4. The small shop has about 4 orders a month. December is €22.70, October €255. What follows?

Nothing yet: 85.6% of the made-up years in which all months are the same show such a gapOctober was strong because of the adsDecember was a weak month

Check

Start the lesson over

Related lessons

• Builds on: Groups: groupby · The first chart · Distributions · Describing the data
• Continues in: Confidence intervals · Testing a hypothesis · A/B test · Cross-validation
• Tasks on this topic: Statistics for decisions