Correlation and cause

In short: Correlation shows that two things move together, not that one causes the other; a third cause or Simpson’s paradox can turn the conclusion around.

Problem

“The banner lifts orders”

Marketing put up a “Free delivery” banner. It was not a test: the banner was shown at someone’s discretion. The log has 10,000 visits: the device, whether the banner was seen (saw_banner), whether there was an order:

Pythondata: the shop tables

import pandas as pd

banner = pd.read_csv('banner_visits.csv')
print(len(banner))
print(banner.head(3))
(banner.groupby('saw_banner')['ordered'].mean() * 100).round(2)

import pandas as pd banner = pd.read_csv('banner_visits.csv') print(len(banner)) print(banner.head(3)) (banner.groupby('saw_banner')['ordered'].mean() * 100).round(2)

▶ RunRestore the code

With the banner 5.66% of visits order, without it 3.63%. Marketing wants to show it to everyone: the gap is 2 points, and with the banner there are 56% more orders.

But does the banner cause it?

The idea

Who saw the banner?

Before you trust the gap, look at who is in each group:

Pythondata: the shop tables

import pandas as pd

banner = pd.read_csv('banner_visits.csv')
print(pd.crosstab(banner['device'], banner['saw_banner']))
(banner.groupby('device')['ordered'].mean() * 100).round(2)

import pandas as pd banner = pd.read_csv('banner_visits.csv') print(pd.crosstab(banner['device'], banner['saw_banner'])) (banner.groupby('device')['ordered'].mean() * 100).round(2)

▶ RunRestore the code

Of 3000 visits from a desktop, 2400 saw the banner. Of 7000 from a phone, only 1400. And desktops order more often anyway: 7.70% against 2.99%.

The “with banner” group is full of desktops, and the “without banner” group of phones. So you are not comparing banner with no banner, but mostly desktop with phone. The device is a confounder: it affects both who sees the banner and who orders.

The idea

Split by device

Compare the banner within each device. Then the device is the same on both sides:

Pythondata: the shop tables

import pandas as pd

banner = pd.read_csv('banner_visits.csv')
rates = banner.pivot_table(index='device', columns='saw_banner', values='ordered', aggfunc='mean') * 100
rates.round(2)

import pandas as pd banner = pd.read_csv('banner_visits.csv') rates = banner.pivot_table(index='device', columns='saw_banner', values='ordered', aggfunc='mean') * 100 rates.round(2)

▶ RunRestore the code

On desktop 7.50% order with the banner, 8.50% without it. On mobile 2.50% against 3.11%. The banner does worse on both devices, while overall it looked better. This is called Simpson’s paradox: the direction for everyone together is the opposite of the direction in each group.

More precisely, if you are curious

On each device on its own the difference is within the noise: 600 desktop visits without the banner are few.

Pythondata: the shop tables

import pandas as pd

banner = pd.read_csv('banner_visits.csv')
from scipy import stats

for device in ['desktop', 'mobile']:
    part = banner[banner['device'] == device]
    table = pd.crosstab(part['saw_banner'], part['ordered'])
    print(device, round(stats.chi2_contingency(table).pvalue, 2))

import pandas as pd banner = pd.read_csv('banner_visits.csv') from scipy import stats for device in ['desktop', 'mobile']: part = banner[banner['device'] == device] table = pd.crosstab(part['saw_banner'], part['ordered']) print(device, round(stats.chi2_contingency(table).pvalue, 2))

▶ RunRestore the code

So the honest conclusion is not “the banner hurts” but “there is no evidence that the banner helps”. The opposite direction overall comes entirely from the mix of devices.

Try it

Compute the paradox

Store in gap the difference in conversion with banner − without banner for each device, in points, and in overall_gap the same difference for all visits together.

Pythondata: the shop tables

import pandas as pd

banner = pd.read_csv('banner_visits.csv')
rates = ...
gap = ...
overall_gap = ...
print(gap)
print(overall_gap)

import pandas as pd banner = pd.read_csv('banner_visits.csv') rates = ... gap = ... overall_gap = ... print(gap) print(overall_gap)

Check▶ Run

The idea

Why the total flips

The overall conversion of a group is an average of the devices, weighted by how many visits from each device are in the group:

Pythondata: the shop tables

import pandas as pd

banner = pd.read_csv('banner_visits.csv')
pd.crosstab(banner['saw_banner'], banner['device'], normalize='index').round(2)

import pandas as pd banner = pd.read_csv('banner_visits.csv') pd.crosstab(banner['saw_banner'], banner['device'], normalize='index').round(2)

▶ RunRestore the code

With the banner 63% of visits are from a desktop, without it only 10%. So the banner group gets a lot of the desktops’ high conversion: (180 + 35) orders from 3800 visits are 5.66%. The group without the banner is almost all phones: (51 + 174) from 6200 are 3.63%.

The overall gap does not come from the banner. It comes from where the banner was shown: where people order more anyway.

The idea

Correlation: corr

The same mistake lurks in the numbers for the year. Here are the units sold by week: drinks (mineral water and orange juice) and vegetables (tomatoes and cucumbers). How much do they go together?

Pythondata: the shop tables

import matplotlib.pyplot as plt

import pandas as pd

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
sold = orders[orders['status'] != 'cancelled']
items = items.merge(sold[['id', 'ordered_at']], left_on='order_id', right_on='id').set_index('ordered_at')
drinks = items[items['product_id'].isin([15, 16])]['quantity'].resample('W').sum()
veg = items[items['product_id'].isin([12, 13])]['quantity'].resample('W').sum()
weekly = pd.DataFrame({'drinks': drinks, 'veg': veg}).iloc[1:-1]
print(len(weekly), 'weeks')
print(round(weekly['drinks'].corr(weekly['veg']), 2))

fig, ax = plt.subplots(figsize=(5, 4))
ax.scatter(weekly['veg'], weekly['drinks'])
ax.set_title('Drinks and vegetables by week')
ax.set_xlabel('vegetables, units')
ax.set_ylabel('drinks, units')
fig.tight_layout()

import matplotlib.pyplot as plt import pandas as pd orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') sold = orders[orders['status'] != 'cancelled'] items = items.merge(sold[['id', 'ordered_at']], left_on='order_id', right_on='id').set_index('ordered_at') drinks = items[items['product_id'].isin([15, 16])]['quantity'].resample('W').sum() veg = items[items['product_id'].isin([12, 13])]['quantity'].resample('W').sum() weekly = pd.DataFrame({'drinks': drinks, 'veg': veg}).iloc[1:-1] print(len(weekly), 'weeks') print(round(weekly['drinks'].corr(weekly['veg']), 2)) fig, ax = plt.subplots(figsize=(5, 4)) ax.scatter(weekly['veg'], weekly['drinks']) ax.set_title('Drinks and vegetables by week') ax.set_xlabel('vegetables, units') ax.set_ylabel('drinks, units') fig.tight_layout()

▶ RunRestore the code

The correlation is a number from −1 to 1. Close to 1: when one is large, the other is large too, and the points lie around a line going up. Around 0: there is no such link. Close to −1: one grows, the other falls. a.corr(b) computes it, and df.corr() gives all pairs of columns at once.

Here it is 0.46: in the weeks with more vegetables, more drinks are sold too. Does that mean vegetables sell drinks?

The idea

The third thing: the season

Split the weeks into summer (June to August) and the rest:

Pythondata: the shop tables

import matplotlib.pyplot as plt

import pandas as pd

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
sold = orders[orders['status'] != 'cancelled']
items = items.merge(sold[['id', 'ordered_at']], left_on='order_id', right_on='id').set_index('ordered_at')
drinks = items[items['product_id'].isin([15, 16])]['quantity'].resample('W').sum()
veg = items[items['product_id'].isin([12, 13])]['quantity'].resample('W').sum()
weekly = pd.DataFrame({'drinks': drinks, 'veg': veg}).iloc[1:-1]
summer = weekly.index.month.isin([6, 7, 8])
print(weekly.groupby(summer).mean().round(1))
print('summer:', round(weekly[summer]['drinks'].corr(weekly[summer]['veg']), 2))
print('the rest:', round(weekly[~summer]['drinks'].corr(weekly[~summer]['veg']), 2))

fig, ax = plt.subplots(figsize=(5, 4))
ax.scatter(weekly[~summer]['veg'], weekly[~summer]['drinks'], color='grey', label='the other weeks')
ax.scatter(weekly[summer]['veg'], weekly[summer]['drinks'], color='tab:orange', label='June to August')
ax.set_title('Summer lifts both')
ax.set_xlabel('vegetables, units')
ax.set_ylabel('drinks, units')
ax.legend()
fig.tight_layout()

import matplotlib.pyplot as plt import pandas as pd orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') sold = orders[orders['status'] != 'cancelled'] items = items.merge(sold[['id', 'ordered_at']], left_on='order_id', right_on='id').set_index('ordered_at') drinks = items[items['product_id'].isin([15, 16])]['quantity'].resample('W').sum() veg = items[items['product_id'].isin([12, 13])]['quantity'].resample('W').sum() weekly = pd.DataFrame({'drinks': drinks, 'veg': veg}).iloc[1:-1] summer = weekly.index.month.isin([6, 7, 8]) print(weekly.groupby(summer).mean().round(1)) print('summer:', round(weekly[summer]['drinks'].corr(weekly[summer]['veg']), 2)) print('the rest:', round(weekly[~summer]['drinks'].corr(weekly[~summer]['veg']), 2)) fig, ax = plt.subplots(figsize=(5, 4)) ax.scatter(weekly[~summer]['veg'], weekly[~summer]['drinks'], color='grey', label='the other weeks') ax.scatter(weekly[summer]['veg'], weekly[summer]['drinks'], color='tab:orange', label='June to August') ax.set_title('Summer lifts both') ax.set_xlabel('vegetables, units') ax.set_ylabel('drinks, units') ax.legend() fig.tight_layout()

▶ RunRestore the code

In summer, on average 121.6 drinks and 68.2 vegetables sell per week, the rest of the time 81.5 and 38.5. Summer lifts both. Within the summer, though, the correlation is −0.13, and in the other weeks −0.15: about zero. The link comes from the season, not from the vegetables.

1. In the weeks when more tomatoes sell, more mineral water sells too. A colleague suggests: “Let’s cut back on tomatoes, to sell more water.” What is he missing?

The correlation is too small to countNothing: the correlation is positive, so tomatoes sell waterA third factor, the season, lifts both: tomatoes do not make anyone buy water

Check

The idea

Why the A/B test is different

In the previous lesson the new cart was better, and that was a cause, not just a link. The difference from the banner is one thing: who decides who sees what.

• In the A/B test chance decides. In both groups desktops were 35%. Everything you do not measure is spread evenly as well.
• With the banner someone decides. They show it mostly on desktop, and the groups differ in the device, and maybe in other things that are not in the table.

Splitting by device fixes only the factor you know about and have recorded. Random assignment fixes the ones you do not know about too. That is why, when you can, you run a test.

The idea

How to say it honestly

Data that were only collected, not from a test, are called observational. From them you can say “they go together”, not “one leads to the other”. Compare:

• “The banner lifts orders by 2 points.” This is a claim about a cause, which the data do not give.
• “Visits with the banner order more often, but mostly because they are from a desktop. On each device on its own the banner does not help. To answer this we need an A/B test.”

Before you trust a difference in observational data, ask:

1. Who ends up in each group, and why?
2. Is there a third thing (device, season, type of customer) that moves both?
3. Does the difference flip if you split by it?

Exercise

A fair comparison

Compare the banner fairly: what is the conversion with and without the banner if both groups had the device mix of all visits? Store in desktop_share the share of desktops among all visits, and in with_banner and without_banner the weighted conversions in percent.

Pythondata: the shop tables

import pandas as pd

banner = pd.read_csv('banner_visits.csv')
rates = banner.pivot_table(index='device', columns='saw_banner', values='ordered', aggfunc='mean') * 100
desktop_share = ...
with_banner = ...
without_banner = ...
print(with_banner, without_banner)

import pandas as pd banner = pd.read_csv('banner_visits.csv') rates = banner.pivot_table(index='device', columns='saw_banner', values='ordered', aggfunc='mean') * 100 desktop_share = ... with_banner = ... without_banner = ... print(with_banner, without_banner)

Check▶ Run

The gift card and the pan sell together: weeks with many cards have many pans too. Store in overall their correlation over all weeks, and in outside the correlation without the holiday weeks (the holidays mask: from 1 December to 4 January).

Pythondata: the shop tables

import pandas as pd

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
sold = orders[orders['status'] != 'cancelled']
items = items.merge(sold[['id', 'ordered_at']], left_on='order_id', right_on='id').set_index('ordered_at')
gift = items[items['product_id'] == 25]['quantity'].resample('W').sum()
pan = items[items['product_id'] == 22]['quantity'].resample('W').sum()
weekly = pd.DataFrame({'gift': gift, 'pan': pan}).fillna(0).iloc[1:-1]
holidays = (weekly.index >= '2025-12-01') & (weekly.index <= '2026-01-04')
overall = ...
outside = ...
print(overall, outside)

import pandas as pd orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') sold = orders[orders['status'] != 'cancelled'] items = items.merge(sold[['id', 'ordered_at']], left_on='order_id', right_on='id').set_index('ordered_at') gift = items[items['product_id'] == 25]['quantity'].resample('W').sum() pan = items[items['product_id'] == 22]['quantity'].resample('W').sum() weekly = pd.DataFrame({'gift': gift, 'pan': pan}).fillna(0).iloc[1:-1] holidays = (weekly.index >= '2025-12-01') & (weekly.index <= '2026-01-04') overall = ... outside = ... print(overall, outside)

Check▶ Run

Test

Check yourself

1. What is Simpson’s paradox?

When the correlation is negativeA difference that points one way for everyone together and the other way in every group on its ownWhen p is under 0.05 and the difference is small

2. Why is the device a confounder for the banner?

Because there are more phonesBecause the banner looks bad on a phoneIt affects both who sees the banner (mostly desktops) and who orders (desktops order more)

3. The correlation between two columns is 0.46. What do you know for sure?

One causes the otherWhen one is larger, the other is more often larger too46% of the change in one comes from the other

4. Why does the A/B test of the cart not suffer from the banner’s problem?

Because only the conversion is measuredBecause there are more visitsThe variant was assigned at random, so the devices are spread equally across both groups

5. Which sentence about the banner log is honest?

“The banner lowers orders, proven.”“The banner lifts orders by 2 points.”“Visits with the banner order more often, but mostly because they are from a desktop. On each device the banner does not help.”

Check

Start the lesson over

Related lessons

• Builds on: The first chart · Distributions · Testing a hypothesis · A/B test
• Continues in: Linear regression · Why did revenue fall?
• Tasks on this topic: Statistics for decisions