Distributions

In short: A distribution shows how all the values are spread at once: a histogram and a box plot show it for one variable, and a scatter plot shows the link between two.

Problem

“How much is an order worth?”

The manager wants to know what one order is worth. The mean has been worked out: €10.63. The advertising will be planned for “the typical customer with an order of about €10”.

Pythondata: the shop tables

import pandas as pd
import matplotlib.pyplot as plt

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
per_order = items.groupby('order_id', as_index=False).agg(
    revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum'))
orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one')
# Revenue leaves out the cancelled orders.
sold = orders[orders['status'] != 'cancelled']
values = sold['revenue']
average = values.mean()
print(len(sold))
print(round(average, 2))
print((values < average).sum())

import pandas as pd import matplotlib.pyplot as plt orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] per_order = items.groupby('order_id', as_index=False).agg( revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum')) orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one') # Revenue leaves out the cancelled orders. sold = orders[orders['status'] != 'cancelled'] values = sold['revenue'] average = values.mean() print(len(sold)) print(round(average, 2)) print((values < average).sum())

▶ RunRestore the code

There are 10,459 sold orders (without the cancelled ones), and 6,736 of them are below the mean. Almost two in every three. How can the “average” be above most of them? You see the answer when you look at all the orders at once instead of one number.

The idea

Histogram

How the values spread out, which are common and which are rare, is called a distribution. Its picture is the histogram: the x axis is split into equal intervals, and each bar is how many orders fall into its interval.

ax.hist(values, bins=30) makes 30 intervals from the smallest to the largest value and counts them. It returns three things: the count in each interval, the edges and the bars themselves.

Pythondata: the shop tables

import pandas as pd
import matplotlib.pyplot as plt

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
per_order = items.groupby('order_id', as_index=False).agg(
    revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum'))
orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one')
# Revenue leaves out the cancelled orders.
sold = orders[orders['status'] != 'cancelled']
values = sold['revenue']
fig, ax = plt.subplots()
counts, edges, bars = ax.hist(values, bins=30)
ax.set_title('Order values for the year')
ax.set_xlabel('Order value, €')
ax.set_ylabel('Number of orders')
print(edges[:3].round(2))
print(counts[:2].astype(int))

import pandas as pd import matplotlib.pyplot as plt orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] per_order = items.groupby('order_id', as_index=False).agg( revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum')) orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one') # Revenue leaves out the cancelled orders. sold = orders[orders['status'] != 'cancelled'] values = sold['revenue'] fig, ax = plt.subplots() counts, edges, bars = ax.hist(values, bins=30) ax.set_title('Order values for the year') ax.set_xlabel('Order value, €') ax.set_ylabel('Number of orders') print(edges[:3].round(2)) print(counts[:2].astype(int))

▶ RunRestore the code

The first bar runs from €0.80 to €3.40 and holds 1,915 orders. The second, from €3.40 to €5.99, is the highest: 2,122. After that the bars fall, and on the right a thin strip of rare orders remains, all the way to €78.70. That strip is called the tail. A distribution with a long tail to the right is normal for money: many small purchases and a few large ones.

The idea

The number of bins changes the picture

You choose bins, and a different number tells a different story. Here are the same orders with 4 and with 80 bins:

Pythondata: the shop tables

import pandas as pd
import matplotlib.pyplot as plt

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
per_order = items.groupby('order_id', as_index=False).agg(
    revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum'))
orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one')
# Revenue leaves out the cancelled orders.
sold = orders[orders['status'] != 'cancelled']
values = sold['revenue']
fig, (left, right) = plt.subplots(1, 2, figsize=(10, 4))
few, few_edges, _ = left.hist(values, bins=4)
left.set_title('4 bins')
right.hist(values, bins=80)
right.set_title('80 bins')
fig.suptitle('The same orders, a different number of bins')
for ax in (left, right):
    ax.set_xlabel('Order value, €')
left.set_ylabel('Number of orders')
fig.tight_layout()
print(few_edges.round(2))
print(few.astype(int))

import pandas as pd import matplotlib.pyplot as plt orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] per_order = items.groupby('order_id', as_index=False).agg( revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum')) orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one') # Revenue leaves out the cancelled orders. sold = orders[orders['status'] != 'cancelled'] values = sold['revenue'] fig, (left, right) = plt.subplots(1, 2, figsize=(10, 4)) few, few_edges, _ = left.hist(values, bins=4) left.set_title('4 bins') right.hist(values, bins=80) right.set_title('80 bins') fig.suptitle('The same orders, a different number of bins') for ax in (left, right): ax.set_xlabel('Order value, €') left.set_ylabel('Number of orders') fig.tight_layout() print(few_edges.round(2)) print(few.astype(int))

▶ RunRestore the code

With 4 bins you only see that 9,091 orders are below €20.28, 1,182 are up to €39.75, and after that there are hardly any. With 80 bins you see that most orders are between €2 and €8, and the count falls off smoothly towards the tail. The spikes (one bar high, its neighbour low) come from the prices: some totals add up more often than others.

There is no single right number. Try a few and pick the one where the shape shows and the noise does not hide it. For a few thousand values, 30–60 bins is usually a good start.

More precisely, if you are curious

bins can also be a list of edges: bins=[0, 5, 10, 20, 40, 80]. Then the bins are not equally wide, and the height misleads: a wide bin collects more orders only because it is wide. With density=True the height is divided by the width of the bin, and the wide bins no longer mislead.

Try it

December up close

What do the December orders look like? Store the sold orders from December in december and draw a histogram of their value with 40 bins. Label the x axis.

Pythondata: the shop tables

import pandas as pd
import matplotlib.pyplot as plt

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
per_order = items.groupby('order_id', as_index=False).agg(
    revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum'))
orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one')
# Revenue leaves out the cancelled orders.
sold = orders[orders['status'] != 'cancelled']
values = sold['revenue']
december = sold[...]
fig, ax = plt.subplots()
ax.hist(..., bins=...)
ax.set_title('Orders in December')

import pandas as pd import matplotlib.pyplot as plt orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] per_order = items.groupby('order_id', as_index=False).agg( revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum')) orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one') # Revenue leaves out the cancelled orders. sold = orders[orders['status'] != 'cancelled'] values = sold['revenue'] december = sold[...] fig, ax = plt.subplots() ax.hist(..., bins=...) ax.set_title('Orders in December')

Check▶ Run

The idea

Mean and median

Sort all the orders from the smallest to the largest. The value right in the middle is the median: half the orders are below it, half above. values.median() gives it.

Draw the mean and the median on the histogram as vertical lines. ax.axvline(x) draws a line through the whole height of the chart at the point x:

Pythondata: the shop tables

import pandas as pd
import matplotlib.pyplot as plt

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
per_order = items.groupby('order_id', as_index=False).agg(
    revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum'))
orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one')
# Revenue leaves out the cancelled orders.
sold = orders[orders['status'] != 'cancelled']
values = sold['revenue']
average = values.mean()
middle = values.median()
fig, ax = plt.subplots()
ax.hist(values, bins=60, color='lightgray')
ax.axvline(average, color='black', label='mean')
ax.axvline(middle, color='black', linestyle='--', label='median')
ax.set_title('The mean and the median of the orders')
ax.set_xlabel('Order value, €')
ax.set_ylabel('Number of orders')
ax.legend()
print(round(average, 2), round(middle, 2))

import pandas as pd import matplotlib.pyplot as plt orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] per_order = items.groupby('order_id', as_index=False).agg( revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum')) orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one') # Revenue leaves out the cancelled orders. sold = orders[orders['status'] != 'cancelled'] values = sold['revenue'] average = values.mean() middle = values.median() fig, ax = plt.subplots() ax.hist(values, bins=60, color='lightgray') ax.axvline(average, color='black', label='mean') ax.axvline(middle, color='black', linestyle='--', label='median') ax.set_title('The mean and the median of the orders') ax.set_xlabel('Order value, €') ax.set_ylabel('Number of orders') ax.legend() print(round(average, 2), round(middle, 2))

▶ RunRestore the code

The mean is €10.63, and the median is €7.70. The median sits where the bulk of the bars is. The mean is to its right: the tail of large orders pulls it up, because every euro of them goes into the sum. The median does not care whether the largest order is €78 or €7,800.

So with money, times and sizes, where there is a tail, look at both. If they are far apart, the median is closer to “the typical customer”.

1. One order for €5,000 joins the 10,459 sold orders (a company buys gifts). What happens to the mean (€10.63) and to the median (€7.70)?

The mean becomes €11.11, and the median stays €7.70The median jumps to €5,000, and the mean staysBoth grow by about €0.48

Check

The idea

Box plot

A histogram is detailed, but two or five histograms side by side are hard to compare. A box plot shrinks a distribution to five things. December is the strongest month: are its orders larger, or are there only more of them?

Pythondata: the shop tables

import pandas as pd
import matplotlib.pyplot as plt

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
per_order = items.groupby('order_id', as_index=False).agg(
    revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum'))
orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one')
# Revenue leaves out the cancelled orders.
sold = orders[orders['status'] != 'cancelled']
values = sold['revenue']
month = sold['ordered_at'].dt.month
december = sold[month == 12]['revenue']
rest = sold[month != 12]['revenue']
fig, ax = plt.subplots()
ax.boxplot([december, rest], tick_labels=['December', 'other months'])
ax.set_title('Order values')
ax.set_ylabel('€ per order')
print(round(december.median(), 2), round(rest.median(), 2))
print(round(december.mean(), 2), round(rest.mean(), 2))

import pandas as pd import matplotlib.pyplot as plt orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] per_order = items.groupby('order_id', as_index=False).agg( revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum')) orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one') # Revenue leaves out the cancelled orders. sold = orders[orders['status'] != 'cancelled'] values = sold['revenue'] month = sold['ordered_at'].dt.month december = sold[month == 12]['revenue'] rest = sold[month != 12]['revenue'] fig, ax = plt.subplots() ax.boxplot([december, rest], tick_labels=['December', 'other months']) ax.set_title('Order values') ax.set_ylabel('€ per order') print(round(december.median(), 2), round(rest.median(), 2)) print(round(december.mean(), 2), round(rest.mean(), 2))

▶ RunRestore the code

How to read it:

• the line in the box is the median: €8.60 in December and €7.60 in the other months;
• the box is the middle half of the orders: a quarter are below it, a quarter above;
• the whiskers reach the last usual values;
• the circles are unusually large orders.

The December box is taller and reaches higher: there are more large orders there (gifts, pans, knives). So the mean changes more than the median: €13.14 against €10.30, while the median is €8.60 against €7.60. December has more orders and larger ones, but the difference is mostly in the tail.

More precisely, if you are curious

A whisker reaches the farthest order that is no more than 1.5 box heights from the box. Anything farther is a circle. You will compute this rule and the box height (IQR) with numbers in the lesson on describing data.

In older versions of matplotlib (before 3.9) the labels are given with labels= instead of tick_labels=.

The idea

Scatter: two numbers per order

Does the money grow with the number of lines in the order? Here you have two numbers per order, and the question is about how they relate. The picture is a scatter plot: each order is a point, x is one number, y is the other. ax.scatter(x, y) draws it.

Pythondata: the shop tables

import pandas as pd
import matplotlib.pyplot as plt

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
per_order = items.groupby('order_id', as_index=False).agg(
    revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum'))
orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one')
# Revenue leaves out the cancelled orders.
sold = orders[orders['status'] != 'cancelled']
values = sold['revenue']
fig, (left, right) = plt.subplots(1, 2, figsize=(10, 4), sharey=True)
left.scatter(sold['lines'], sold['revenue'])
left.set_title('No transparency')
right.scatter(sold['lines'], sold['revenue'], alpha=0.03)
right.set_title('alpha=0.03')
fig.suptitle('Lines and order value')
for ax in (left, right):
    ax.set_xlabel('Lines in the order')
left.set_ylabel('Order value, €')
fig.tight_layout()
sold.groupby('lines')['revenue'].agg(['size', 'mean']).round(2)

import pandas as pd import matplotlib.pyplot as plt orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] per_order = items.groupby('order_id', as_index=False).agg( revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum')) orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one') # Revenue leaves out the cancelled orders. sold = orders[orders['status'] != 'cancelled'] values = sold['revenue'] fig, (left, right) = plt.subplots(1, 2, figsize=(10, 4), sharey=True) left.scatter(sold['lines'], sold['revenue']) left.set_title('No transparency') right.scatter(sold['lines'], sold['revenue'], alpha=0.03) right.set_title('alpha=0.03') fig.suptitle('Lines and order value') for ax in (left, right): ax.set_xlabel('Lines in the order') left.set_ylabel('Order value, €') fig.tight_layout() sold.groupby('lines')['revenue'].agg(['size', 'mean']).round(2)

▶ RunRestore the code

On the left everything is in solid columns. Thousands of points lie on top of each other, and you cannot see where there are many and where there are few. This is called overplotting. On the right alpha=0.03 makes each point almost transparent: where there are many, the colour builds up and it is dark.

Now you see that with each line the order grows by about €4–5 on average: from €4.27 with one line (2,517 orders) to €27.48 with six (145 orders). But the points of each line spread upwards: one expensive product costs as much as ten cheap ones.

sharey=True gives the two charts the same y axis, so that they compare fairly.

Try it

Pieces and value

The pieces column is the number of pieces in the order (the sum of quantity). Draw a scatter plot: the pieces on x, the order value on y, with transparency so you can see where many orders are. Label both axes.

Pythondata: the shop tables

import pandas as pd
import matplotlib.pyplot as plt

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
per_order = items.groupby('order_id', as_index=False).agg(
    revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum'))
orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one')
# Revenue leaves out the cancelled orders.
sold = orders[orders['status'] != 'cancelled']
values = sold['revenue']
fig, ax = plt.subplots()
ax.scatter(..., ...)
ax.set_title('Pieces and order value')

import pandas as pd import matplotlib.pyplot as plt orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] per_order = items.groupby('order_id', as_index=False).agg( revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum')) orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one') # Revenue leaves out the cancelled orders. sold = orders[orders['status'] != 'cancelled'] values = sold['revenue'] fig, ax = plt.subplots() ax.scatter(..., ...) ax.set_title('Pieces and order value')

Check▶ Run

Exercise

The answer for the manager

The manager wants one chart: what the orders are worth and where the mean and the median are. Store the mean of the sold orders' value in mean_value and the median in median_value. Draw a histogram with 50 bins and the two values as vertical lines, with a legend.

Pythondata: the shop tables

import pandas as pd
import matplotlib.pyplot as plt

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['revenue'] = items['quantity'] * items['unit_price']
per_order = items.groupby('order_id', as_index=False).agg(
    revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum'))
orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one')
# Revenue leaves out the cancelled orders.
sold = orders[orders['status'] != 'cancelled']
values = sold['revenue']
mean_value = ...
median_value = ...
fig, ax = plt.subplots()
ax.hist(values, bins=50, color='lightgray')

import pandas as pd import matplotlib.pyplot as plt orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['revenue'] = items['quantity'] * items['unit_price'] per_order = items.groupby('order_id', as_index=False).agg( revenue=('revenue', 'sum'), lines=('product_id', 'size'), pieces=('quantity', 'sum')) orders = orders.merge(per_order, left_on='id', right_on='order_id', how='left', validate='one_to_one') # Revenue leaves out the cancelled orders. sold = orders[orders['status'] != 'cancelled'] values = sold['revenue'] mean_value = ... median_value = ... fig, ax = plt.subplots() ax.hist(values, bins=50, color='lightgray')

Check▶ Run

Test

Check yourself

1. What does one bar of the histogram of order values show?

How many orders have a value in the interval under the barThe value of one orderThe average order value in one month

2. With bins=4 the histogram shows 9,091 orders between €0.80 and €20.28. What do you lose compared with bins=80?

The largest orderThe shape inside the bar: that most orders are between €2 and €8 and the count falls off smoothlyNothing, the number of orders is the same

3. What is the line inside the box of a box plot?

The meanThe most frequent valueThe median

4. A scatter plot with 10,459 points looks like solid blue columns. What helps?

More bins: bins=100A larger chart with figsizeTransparency: alpha=0.05, so the dense parts turn dark and the sparse ones pale

5. The mean order is €10.63, and the median is €7.70. What do the two numbers say together?

There is a long tail of large orders that pulls the mean up; the typical order is closer to €7.70The data has an error, the two should be equalMost orders are above €10.63

Check

Start the lesson over

Related lessons

• Builds on: Groups: groupby · Joining: merge · The first chart · Which chart for which question
• Continues in: Describing the data · Sample and population · Confidence intervals · Correlation and cause
• Tasks on this topic: Charts