Describing the data

In short: Descriptive statistics tell where the centre of the data is (mean, median, most common value) and how far it spreads (standard deviation, percentiles, IQR).

Problem

“The average customer”

The manager wants a loyalty card. “The average customer places 8 orders a year. Let us give a gift with the eighth order: that way every ordinary customer gets it.”

It sounds reasonable. But how many customers reach an eighth order? And how many orders does an ordinary customer place? The mean answers only one of these questions, and not always the one you think.

The idea

Mean, median, most common

One number that says “the orders are about this much” comes in three kinds:

• the mean: the sum divided by the count;
• the median: the number in the middle once you sort them; half are below it, half above;
• the mode: the value that occurs most often.

The order value for the year, without the cancelled orders:

Pythondata: the shop tables

import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
per_order = items.groupby('order_id')['revenue'].sum()
orders['value'] = orders['id'].map(per_order)
# Cancelled orders are not revenue: leave them out.
values = orders[orders['status'] != 'cancelled']['value'].round(2)
print('orders:', len(values))
print('mean:', round(values.mean(), 2))
print('median:', values.median())
print('most common:', values.mode().tolist())
print('below the mean:', round((values < values.mean()).mean() * 100, 1), '%')

import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] per_order = items.groupby('order_id')['revenue'].sum() orders['value'] = orders['id'].map(per_order) # Cancelled orders are not revenue: leave them out. values = orders[orders['status'] != 'cancelled']['value'].round(2) print('orders:', len(values)) print('mean:', round(values.mean(), 2)) print('median:', values.median()) print('most common:', values.mode().tolist()) print('below the mean:', round((values < values.mean()).mean() * 100, 1), '%')

▶ RunRestore the code

The mean is €10.63, the median €7.70. The most common order is €2.40. Three numbers for the same data, and 64.4% of the orders are below the mean.

The reason is the tail: a few big orders (up to €78.70) pull the mean up. The median does not feel them, because it only looks at what is in the middle. You saw the tail in the histogram of the orders.

mode() returns a list because several values may share the top count. That is why values is rounded to the cent: otherwise 2.4 and 2.4000000000000004 count as different values.

1. Five orders: €4, €6, €7, €9 and €12. A sixth one comes in, for €500 (an office orders for the whole year). What happens?

The mean jumps from €7.60 to €89.67, while the median only moves from €7 to €8The median jumps, the mean hardly changesBoth jump by the same amount

Check

The idea

Spread: range and standard deviation

The mean does not say how different the orders are from each other. Two shops with a mean of €10 can be very different: in one every order is about €10, in the other there are orders of €1 and of €50. This is called spread.

Pythondata: the shop tables

import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
per_order = items.groupby('order_id')['revenue'].sum()
orders['value'] = orders['id'].map(per_order)
# Cancelled orders are not revenue: leave them out.
values = orders[orders['status'] != 'cancelled']['value'].round(2)
print('smallest:', values.min(), ' largest:', values.max())
print('range:', round(values.max() - values.min(), 2))
print('standard deviation:', round(values.std(), 2))

import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] per_order = items.groupby('order_id')['revenue'].sum() orders['value'] = orders['id'].map(per_order) # Cancelled orders are not revenue: leave them out. values = orders[orders['status'] != 'cancelled']['value'].round(2) print('smallest:', values.min(), ' largest:', values.max()) print('range:', round(values.max() - values.min(), 2)) print('standard deviation:', round(values.std(), 2))

▶ RunRestore the code

The range is the largest minus the smallest: €77.90. It depends only on the two extreme orders, and one odd order changes it.

The standard deviation (std) is how far from the mean an order typically is. Here it is €9.43: an order of €10.63 ± about €9 is nothing unusual.

More precisely, if you are curious

std is the square root of the mean squared distance from the mean. Squaring makes big distances much heavier, so with a long tail std grows too.

pandas divides by n − 1, NumPy (np.std) by n. With 10,459 orders the difference is in the fourth decimal place. With 5 numbers it shows: np.std(x, ddof=1) gives the same as pandas.

The idea

Percentiles and the IQR

A percentile says below what value a given share of the data lies. The 90th percentile is the value below which 90% of the orders are. quantile gives them with a share from 0 to 1:

Pythondata: the shop tables

import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
per_order = items.groupby('order_id')['revenue'].sum()
orders['value'] = orders['id'].map(per_order)
# Cancelled orders are not revenue: leave them out.
values = orders[orders['status'] != 'cancelled']['value'].round(2)
import matplotlib.pyplot as plt

q = values.quantile([0.1, 0.25, 0.5, 0.75, 0.9])
print(q)
print('IQR:', round(q[0.75] - q[0.25], 2))

fig, ax = plt.subplots()
ax.hist(values, bins=40)
for p in [0.25, 0.5, 0.75]:
    ax.axvline(q[p], color='black', linestyle='--')
ax.set_title('Order value: the quartiles')
ax.set_xlabel('€')
ax.set_ylabel('orders')

import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] per_order = items.groupby('order_id')['revenue'].sum() orders['value'] = orders['id'].map(per_order) # Cancelled orders are not revenue: leave them out. values = orders[orders['status'] != 'cancelled']['value'].round(2) import matplotlib.pyplot as plt q = values.quantile([0.1, 0.25, 0.5, 0.75, 0.9]) print(q) print('IQR:', round(q[0.75] - q[0.25], 2)) fig, ax = plt.subplots() ax.hist(values, bins=40) for p in [0.25, 0.5, 0.75]: ax.axvline(q[p], color='black', linestyle='--') ax.set_title('Order value: the quartiles') ax.set_xlabel('€') ax.set_ylabel('orders')

▶ RunRestore the code

10% of the orders are up to €2.40, and 90% up to €22.92. The three dashed lines are the quartiles: the 25th percentile (q1, €4.20), the median (€7.70) and the 75th (q3, €14.00). They split the orders into four equal parts.

The middle 50% of the orders lie between q1 and q3. Their width is the IQR (interquartile range): 14.00 − 4.20 = €9.80. Unlike the range, the IQR does not move because of one huge order.

Look at the picture too: from the median to q3 it is €6.30, but from q1 to the median only €3.50. The right half is stretched: that is the tail, told in numbers.

Try it

The free delivery threshold

The shop will give free delivery to the largest 10% of orders. Above what amount? Store the threshold in threshold.

Pythondata: the shop tables

import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
per_order = items.groupby('order_id')['revenue'].sum()
orders['value'] = orders['id'].map(per_order)
# Cancelled orders are not revenue: leave them out.
values = orders[orders['status'] != 'cancelled']['value'].round(2)
threshold = ...
print(threshold)

import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] per_order = items.groupby('order_id')['revenue'].sum() orders['value'] = orders['id'].map(per_order) # Cancelled orders are not revenue: leave them out. values = orders[orders['status'] != 'cancelled']['value'].round(2) threshold = ... print(threshold)

Check▶ Run

The idea

describe(), read to the end

describe() from your first look at a table gives everything so far at once:

Pythondata: the shop tables

import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
per_order = items.groupby('order_id')['revenue'].sum()
orders['value'] = orders['id'].map(per_order)
# Cancelled orders are not revenue: leave them out.
values = orders[orders['status'] != 'cancelled']['value'].round(2)
values.describe().round(2)

import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] per_order = items.groupby('order_id')['revenue'].sum() orders['value'] = orders['id'].map(per_order) # Cancelled orders are not revenue: leave them out. values = orders[orders['status'] != 'cancelled']['value'].round(2) values.describe().round(2)

▶ RunRestore the code

Now every row makes sense:

• count: 10,459 orders;
• mean and std: €10.63 on average, typically about €9.43 away from it;
• min and max: from €0.80 to €78.70;
• 25%, 50%, 75%: the quartiles. 50% is the median.

A quick look for a tail: mean is well above 50% (€10.63 against €7.70), and from 75% to max there are €64.70, against €3.40 from min to 25%. The tail is on the right.

The idea

“The average customer” does not exist

Back to the loyalty card. How many orders does each customer place in the year? All orders count, the cancelled ones too: the customer did order.

Pythondata: the shop tables

import pandas as pd

orders = pd.read_csv('year_orders.csv')
per_customer = orders.groupby('customer_id').size()
import matplotlib.pyplot as plt

print('customers:', len(per_customer))
print('mean:', round(per_customer.mean(), 2), ' median:', per_customer.median(), ' most common:', per_customer.mode().tolist())
print('only one order:', (per_customer == 1).sum(), ' most:', per_customer.max())

fig, ax = plt.subplots()
ax.hist(per_customer, bins=60)
ax.axvline(per_customer.mean(), color='black', label='mean')
ax.axvline(per_customer.median(), color='black', linestyle='--', label='median')
ax.set_title('Orders per customer in the year')
ax.set_xlabel('orders')
ax.set_ylabel('customers')
ax.legend()

import pandas as pd orders = pd.read_csv('year_orders.csv') per_customer = orders.groupby('customer_id').size() import matplotlib.pyplot as plt print('customers:', len(per_customer)) print('mean:', round(per_customer.mean(), 2), ' median:', per_customer.median(), ' most common:', per_customer.mode().tolist()) print('only one order:', (per_customer == 1).sum(), ' most:', per_customer.max()) fig, ax = plt.subplots() ax.hist(per_customer, bins=60) ax.axvline(per_customer.mean(), color='black', label='mean') ax.axvline(per_customer.median(), color='black', linestyle='--', label='median') ax.set_title('Orders per customer in the year') ax.set_xlabel('orders') ax.set_ylabel('customers') ax.legend()

▶ RunRestore the code

There are 1390 customers. On average they place 8 orders, but the median is 4 and the most common is 2. 224 customers ordered only once, and the most loyal one has 121 orders.

The histogram shows why: most customers are on the left, at 1–5 orders, and a long tail of regulars reaches 121. The mean (the solid line) sits between the two worlds and describes nobody well. The customer “with 8 orders on average” is a number from a calculation, not a person you will meet often.

When the data has a long tail, the median tells you the ordinary case, and the mean is for something else: for example, how many orders 100 new customers will bring in total.

The idea

Unusual values: the IQR rule

Which orders are unusually big? A common rule: anything more than 1.5 × IQR below q1 or above q3 is unusual.

Pythondata: the shop tables

import pandas as pd

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
per_order = items.groupby('order_id')['revenue'].sum()
orders['value'] = orders['id'].map(per_order)
# Cancelled orders are not revenue: leave them out.
values = orders[orders['status'] != 'cancelled']['value'].round(2)
q1 = values.quantile(0.25)
q3 = values.quantile(0.75)
iqr = q3 - q1
low = q1 - 1.5 * iqr
high = q3 + 1.5 * iqr
print('fences:', round(low, 2), round(high, 2))
print('below the lower:', (values < low).sum(), ' above the upper:', (values > high).sum())
print('the largest:', values[values > high].sort_values().tail(3).tolist())

import pandas as pd orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] per_order = items.groupby('order_id')['revenue'].sum() orders['value'] = orders['id'].map(per_order) # Cancelled orders are not revenue: leave them out. values = orders[orders['status'] != 'cancelled']['value'].round(2) q1 = values.quantile(0.25) q3 = values.quantile(0.75) iqr = q3 - q1 low = q1 - 1.5 * iqr high = q3 + 1.5 * iqr print('fences:', round(low, 2), round(high, 2)) print('below the lower:', (values < low).sum(), ' above the upper:', (values > high).sum()) print('the largest:', values[values > high].sort_values().tail(3).tolist())

▶ RunRestore the code

The upper fence is 14.00 + 1.5 × 9.80 = €28.70, and 609 orders are above it. The lower fence is negative, so nothing is below it.

The rule only points. The decision is yours: these 609 orders are real and bring in about 21% of the revenue. You remove a value only when it is a proven error, like a price without its decimal point or a duplicate, as you saw in the cleaning lesson.

Try it

The regulars

Which customers order unusually often? Compute the upper fence of the IQR rule for orders per customer (in limit) and store the customers above it in regulars.

Pythondata: the shop tables

import pandas as pd

orders = pd.read_csv('year_orders.csv')
per_customer = orders.groupby('customer_id').size()
q1 = per_customer.quantile(0.25)
q3 = ...
limit = ...
regulars = ...
print(limit, len(regulars))

import pandas as pd orders = pd.read_csv('year_orders.csv') per_customer = orders.groupby('customer_id').size() q1 = per_customer.quantile(0.25) q3 = ... limit = ... regulars = ... print(limit, len(regulars))

Check▶ Run

Exercise

A gift with the eighth order?

How many orders does the ordinary customer place? Store it in typical. And what share of the customers reach an eighth order? Store the share (from 0 to 1) in reach.

Pythondata: the shop tables

import pandas as pd

orders = pd.read_csv('year_orders.csv')
per_customer = orders.groupby('customer_id').size()
typical = ...
reach = ...
print(typical, reach)

import pandas as pd orders = pd.read_csv('year_orders.csv') per_customer = orders.groupby('customer_id').size() typical = ... reach = ... print(typical, reach)

Check▶ Run

Test

Check yourself

1. The mean order is €10.63 and the median is €7.70. What does that say about the orders?

Half of the orders are above €10.63One of the calculations is wrongThey have a long tail to the right: a few big orders pull the mean up

2. What is the IQR?

The difference between the third and the first quartile: the width of the middle half of the dataThe difference between the largest and the smallest valueThe average distance from the mean

3. describe() gives 25% 4.20 for the order value. What does it mean?

25% of the revenue comes from orders up to €4.20A quarter of the orders are €4.20 or lessThe orders of €4.20 are 25%

4. The IQR rule finds 609 orders above €28.70. What do you do with them?

You look at them: if they are real (an office, a holiday), they stay in the dataYou remove them because they are errorsYou replace them with the mean

Check

Start the lesson over

Related lessons

• Builds on: New columns · Groups: groupby · The first chart · Distributions
• Continues in: Sample and population · Confidence intervals · Testing a hypothesis · Data leakage
• Tasks on this topic: Statistics for decisions