Testing a hypothesis
In short: A hypothesis test asks whether a difference could be chance: the null hypothesis says “no difference”, and the p-value is how often chance gives a difference at least that big.
Problem
“Is Friday really stronger?”
The manager wants to put one more person on packing on Friday and take one away on Sunday. Before that, they ask: is Friday really stronger, or did it just turn out that way this year? Here is how many orders come in per day on average, by day of the week (0 is Monday, 6 is Sunday):
We count all the orders, the cancelled ones too: the question is how many arrive, and a cancelled order is also received and handled. On a Friday 36.35 orders come in on average, and on a Sunday 24.62. There are 52 Fridays in the year and 52 Sundays too. Every day is a little different: one Friday brings 22 orders, another 68. Is a gap of 11.73 orders real, or is it noise like in the lesson on samples?
The idea
If the day did not matter
The question “is Friday stronger than Sunday?” is a hypothesis. To check it, you start from the boring explanation: the day does not matter and the gap is chance. This is the null hypothesis.
If the null hypothesis is true, the labels “Friday” and “Sunday” mean nothing. You can shuffle them: put the 104 days on one pile, shuffle them and say that the first 52 are “Fridays” and the other 52 “Sundays”. The gap between the two halves is how much gap chance alone gives:
The real gap is 11.73 orders. Three shuffles give 0.50, 1.23 and 1.46. np.concatenate glues the two arrays into one, and rng.permutation returns a shuffled copy.
The idea
10,000 shuffles and the p-value
Three shuffles are too few. You make 10,000 at once: np.tile repeats the 104 days in 10,000 rows, and rng.permuted(..., axis=1) shuffles every row on its own. Then one calculation gives the gap for every row:
The biggest gap out of 10,000 shuffles is 7.50 orders. The real one is 11.73. The share of shuffles with a gap at least this big is the p-value: here it is 0, meaning under 1 in 10,000. If the day did not matter, such a gap would almost never come up. The conclusion: Friday really is stronger.
This approach is called a permutation test. np.abs is there because the question is “is there a difference”, without saying in advance in which direction. The usual threshold is 0.05: below it you say that the difference is statistically significant.
Try it
Monday and Saturday
The gap between Monday and Saturday is smaller. Is it chance? Put in gap the difference in the average number of orders per day (Monday minus Saturday), in gaps the gaps from 10,000 shuffles, and in p the p-value.
Solution
import numpy as np
import pandas as pd
orders = pd.read_csv('year_orders.csv')
orders['ordered_at'] = pd.to_datetime(orders['ordered_at'])
daily = orders.set_index('ordered_at').resample('D').size()
weekday = daily.index.dayofweek.rename('weekday')
mon = daily[weekday == 0].to_numpy()
sat = daily[weekday == 5].to_numpy()
gap = mon.mean() - sat.mean()
both = np.concatenate([mon, sat])
rng = np.random.default_rng(42)
shuffled = rng.permuted(np.tile(both, (10_000, 1)), axis=1)
gaps = shuffled[:, :len(mon)].mean(axis=1) - shuffled[:, len(mon):].mean(axis=1)
p = (np.abs(gaps) >= abs(gap)).mean()
print(round(gap, 2), p)Monday has on average 3.46 orders more than Saturday. Out of 10,000 shuffles about 1% (p = 0.0104) give a gap at least this big. That is under 0.05: there is evidence that Saturday is weaker. There are 52 Mondays and 53 Saturdays, so you cut at len(mon), not at 52. Without rng.permuted all the rows are the same and p is 1, and without np.abs you only count one direction.
The idea
When there is no difference
Sunday has fewer orders. Are the orders themselves smaller too? Here 10,459 orders without the cancelled ones are compared, not days, so a 2,000 × 10,459 table of numbers would take about 170 MB. A loop with 2,000 shuffles of the mask takes under a second:
There are 1,204 Sunday orders, with an average of €10.83, and the others average €10.60. The gap is €0.23 and p = 0.425: in 42.5% of the shuffles the gap is at least this big. This is ordinary noise.
The honest conclusion is “there is no evidence that the Sunday order is different”. It is not “we proved they are equal”: there may be a small difference, only this data cannot show it. On Sunday people order less often, and there is no evidence that they order less.
The idea
What p does not say
The p-value answers one narrow question: how surprising the data is if the null hypothesis is true. It is not two things it is often confused with:
- it is not the probability that the hypothesis is true: p = 0.425 does not mean “a 42.5% chance there is no difference”;
- it is not the size of the difference: a very small p means “not chance”, not “a big difference”.
How big the difference is, the gap itself with an interval tells you. With the bootstrap from the previous lesson, separately for the Fridays and the Sundays:
Friday has on average 11.73 orders more, with a 95% interval from 9.00 to 14.56. That is 1.48 times as many as Sunday. With this number the manager decides how many people to put on, and p only says that they are not chasing noise.
More precisely, if you are curious
The 0.05 threshold is a convention, not a law of nature. At p = 0.06 the data is not “almost evidence”, and at p = 0.04 it is not “surely true”. With a lot of data a tiny difference gives a small p too: with a million orders in each group and a gap of 10 cents the test will be “significant”, but no decision follows from it. So always write both the gap and the interval.
The idea
The same with scipy
scipy.stats has the tests ready. ttest_ind (the t-test) compares the means of two groups by a formula. permutation_test does the shuffling for you, in chunks (batch), so that the memory does not run out:
For the Sunday order the t-test gives 0.426, permutation_test 0.419, and our loop 0.425. The answer is the same: no evidence. For Friday against Sunday the t-test gives 1.04 × 10⁻¹²: zero for all practical purposes, like our 0 out of 10,000.
More precisely, if you are curious
The t-test assumes that the mean of each group is distributed almost normally. With hundreds of orders in a group that is true (the central limit theorem). With little data and a long tail the permutation test is safer, because it assumes nothing about the distribution. permutation_test gives a p slightly different from ours: the shuffles are different, and it also counts the data itself as one of them, (count + 1) / (shuffles + 1).
Exercise
December and Friday
In December gift cards sell. Is the December order bigger than the orders in the other months, or is it chance? Put in gap the difference in the average order (December minus the other months), and in p the p-value. You can use a permutation test or stats.ttest_ind.
Solution
from scipy import stats
import numpy as np
import pandas as pd
orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
orders = orders[orders['status'] != 'cancelled']
items['revenue'] = items['quantity'] * items['unit_price']
totals = items.groupby('order_id')['revenue'].sum()
orders['value'] = orders['id'].map(totals)
orders['ordered_at'] = pd.to_datetime(orders['ordered_at'])
value = orders['value'].to_numpy()
is_dec = (orders['ordered_at'].dt.month == 12).to_numpy()
gap = value[is_dec].mean() - value[~is_dec].mean()
p = stats.ttest_ind(value[is_dec], value[~is_dec]).pvalue
print(is_dec.sum(), round(gap, 2), p)There are 1,211 December orders, and the order is on average €2.84 bigger than the order in the other months. The t-test gives p = 5.1 × 10⁻²³: chance does not explain this gap. The comparison is December against the other 11 months. Against the whole year you would compare December with itself too, and the gap shrinks.
The Friday order is a little smaller than the order on the other days. Is there evidence for that? Run a permutation test with 2,000 shuffles (seed 42): gap is the difference in the average order (Friday minus the other days), p is the p-value, and evidence is whether p is under the 0.05 threshold.
Solution
import numpy as np
import pandas as pd
orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
orders = orders[orders['status'] != 'cancelled']
items['revenue'] = items['quantity'] * items['unit_price']
totals = items.groupby('order_id')['revenue'].sum()
orders['value'] = orders['id'].map(totals)
orders['ordered_at'] = pd.to_datetime(orders['ordered_at'])
value = orders['value'].to_numpy()
is_fri = (orders['ordered_at'].dt.dayofweek == 4).to_numpy()
gap = value[is_fri].mean() - value[~is_fri].mean()
rng = np.random.default_rng(42)
gaps = np.empty(2_000)
for i in range(2_000):
s = rng.permutation(is_fri)
gaps[i] = value[s].mean() - value[~s].mean()
p = (np.abs(gaps) >= abs(gap)).mean()
evidence = p < 0.05
print(round(gap, 2), p, evidence)The Friday order is €0.45 smaller, and p = 0.0645. That is above 0.05: there is no evidence at the usual threshold, and evidence is False. It is tempting to say “almost significant” or to move the threshold to 0.1 after you see p. Do not: the threshold is chosen before the test. The honest answer is “there may be a small difference, but this data does not prove it”.
Test