Cross-validation

In short: Cross-validation splits the data into several folds and trains and checks the model many times, giving a mean score with its spread instead of one lucky number.

Problem

“The tree or the line?”

The shop wants to know how many orders each customer will place over the next 4 months. It uses that to decide whom to send a discount card and how much stock to order. The features come from the orders before 1 July 2026, as in the lesson on classification: number of orders, days since the last one, how long they have been buying and the average order. The target is the number of orders from 1 July to 31 October.

You say “linear regression”, and your colleague says “a tree with 3 levels”. You run them on 6 different splits into training and test (random_state from 0 to 5) and measure the MAE in orders per customer:

Pythondata: the shop tables

import numpy as np
import pandas as pd
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
orders['ordered_at'] = pd.to_datetime(orders['ordered_at'])
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]
customers = before.groupby('customer_id').agg(
    orders=('id', 'size'),
    last=('ordered_at', 'max'),
    first=('ordered_at', 'min'),
    avg_value=('value', 'mean'),
)
customers['days_since'] = (cutoff - customers['last']).dt.days
customers['tenure'] = (cutoff - customers['first']).dt.days
customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0)
X = customers[['orders', 'days_since', 'tenure', 'avg_value']]
y = customers['next']

print(len(X), round(y.mean(), 2))
for seed in range(6):
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=seed)
    line = LinearRegression().fit(X_train, y_train)
    tree = DecisionTreeRegressor(max_depth=3, random_state=0).fit(X_train, y_train)
    line_mae = mean_absolute_error(y_test, line.predict(X_test))
    tree_mae = mean_absolute_error(y_test, tree.predict(X_test))
    print(seed, f'line {line_mae:.2f}', f'tree {tree_mae:.2f}')

import numpy as np import pandas as pd from sklearn.model_selection import KFold, cross_val_score, train_test_split from sklearn.linear_model import LinearRegression from sklearn.tree import DecisionTreeRegressor from sklearn.dummy import DummyRegressor from sklearn.metrics import mean_absolute_error orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') orders['ordered_at'] = pd.to_datetime(orders['ordered_at']) items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] customers = before.groupby('customer_id').agg( orders=('id', 'size'), last=('ordered_at', 'max'), first=('ordered_at', 'min'), avg_value=('value', 'mean'), ) customers['days_since'] = (cutoff - customers['last']).dt.days customers['tenure'] = (cutoff - customers['first']).dt.days customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0) X = customers[['orders', 'days_since', 'tenure', 'avg_value']] y = customers['next'] print(len(X), round(y.mean(), 2)) for seed in range(6): X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=seed) line = LinearRegression().fit(X_train, y_train) tree = DecisionTreeRegressor(max_depth=3, random_state=0).fit(X_train, y_train) line_mae = mean_absolute_error(y_test, line.predict(X_test)) tree_mae = mean_absolute_error(y_test, tree.predict(X_test)) print(seed, f'line {line_mae:.2f}', f'tree {tree_mae:.2f}')

▶ RunRestore the code

There are 1,279 customers, and they place 2.70 orders on average over the 4 months. With splits 0, 1, 2 and 4 the tree is off by less, and with 3 and 5 the line is. Had you run only split 2, your colleague would have won with 1.25 against 1.31. With split 5 you win. Who is right?

The idea

One check is one sample

The test part is 256 customers, picked at random. Like the mean of 30 orders in the lesson on samples, the error on 256 customers is noisy too: another sample gives another number. The error of the line jumps from 1.31 to 1.43 just because of which customers landed in the test.

The gap between the two models is a few hundredths. The noise from the split is around ten hundredths. With one check you cannot say which one is better, just as you cannot tell the average order from 30 orders.

1. With split 0 the tree is off by 1.31 orders and the line by 1.35. What follows from that?

The tree is the better modelThe line is wrongAlmost nothing: with another split of the same customers the line wins

Check

The idea

Cross-validation

The fix is to check many times, with every customer in the test exactly once. You split the customers into 5 folds. You train on 4 and measure on the fifth, then change which one is the test. You get 5 errors for each model. This is called cross-validation.

KFold splits into folds, and cross_val_score trains and measures the model on each. With shuffle=True the customers are shuffled before they are split:

Pythondata: the shop tables

import numpy as np
import pandas as pd
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
orders['ordered_at'] = pd.to_datetime(orders['ordered_at'])
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]
customers = before.groupby('customer_id').agg(
    orders=('id', 'size'),
    last=('ordered_at', 'max'),
    first=('ordered_at', 'min'),
    avg_value=('value', 'mean'),
)
customers['days_since'] = (cutoff - customers['last']).dt.days
customers['tenure'] = (cutoff - customers['first']).dt.days
customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0)
X = customers[['orders', 'days_since', 'tenure', 'avg_value']]
y = customers['next']
kf = KFold(n_splits=5, shuffle=True, random_state=42)
for train, test in kf.split(X):
    print(len(train), len(test))

models = [
    ('mean', DummyRegressor()),
    ('line', LinearRegression()),
    ('tree', DecisionTreeRegressor(max_depth=3, random_state=0)),
]
for name, model in models:
    mae = -cross_val_score(model, X, y, cv=kf, scoring='neg_mean_absolute_error')
    print(name, mae.round(2), f'{mae.mean():.2f} ± {mae.std():.2f}')

import numpy as np import pandas as pd from sklearn.model_selection import KFold, cross_val_score, train_test_split from sklearn.linear_model import LinearRegression from sklearn.tree import DecisionTreeRegressor from sklearn.dummy import DummyRegressor from sklearn.metrics import mean_absolute_error orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') orders['ordered_at'] = pd.to_datetime(orders['ordered_at']) items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] customers = before.groupby('customer_id').agg( orders=('id', 'size'), last=('ordered_at', 'max'), first=('ordered_at', 'min'), avg_value=('value', 'mean'), ) customers['days_since'] = (cutoff - customers['last']).dt.days customers['tenure'] = (cutoff - customers['first']).dt.days customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0) X = customers[['orders', 'days_since', 'tenure', 'avg_value']] y = customers['next'] kf = KFold(n_splits=5, shuffle=True, random_state=42) for train, test in kf.split(X): print(len(train), len(test)) models = [ ('mean', DummyRegressor()), ('line', LinearRegression()), ('tree', DecisionTreeRegressor(max_depth=3, random_state=0)), ] for name, model in models: mae = -cross_val_score(model, X, y, cv=kf, scoring='neg_mean_absolute_error') print(name, mae.round(2), f'{mae.mean():.2f} ± {mae.std():.2f}')

▶ RunRestore the code

Four times you learn on 1,023 customers and check on 256, once on 1,024 and 255. scikit-learn always looks for the biggest number, so the error comes with a minus (neg_mean_absolute_error), and the minus in front turns it back.

The baseline (the mean, DummyRegressor) is off by 2.70 ± 0.09. The line is off by 1.34 ± 0.06, and the tree by 1.38 ± 0.09. The number after ± is the spread of the five errors (std). Both models beat the baseline clearly, and the gap between them (0.04) is smaller than the spread. The honest conclusion is “the line is a little better, but not clearly”. When two models are that close, you take the simpler one.

More precisely, if you are curious

Both models are compared on the same folds: that is why kf is one object, with a random_state. cv=5 also splits into 5, but without shuffling: the first fold is the first 256 customers in order. If the table is sorted (by number, by the date of the first order), the folds differ from each other and the score gets skewed.

Try it

Is the forest better?

Your colleague suggests a random forest (RandomForestRegressor with 50 trees, max_depth=5). Check it with cross-validation on the same folds kf. Store in forest_mae and line_mae the five errors (MAE, positive) of the forest and of the line, in gap the difference in mean error (forest minus line), and in spread the spread of the forest's errors.

Pythondata: the shop tables

from sklearn.ensemble import RandomForestRegressor

import numpy as np
import pandas as pd
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
orders['ordered_at'] = pd.to_datetime(orders['ordered_at'])
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]
customers = before.groupby('customer_id').agg(
    orders=('id', 'size'),
    last=('ordered_at', 'max'),
    first=('ordered_at', 'min'),
    avg_value=('value', 'mean'),
)
customers['days_since'] = (cutoff - customers['last']).dt.days
customers['tenure'] = (cutoff - customers['first']).dt.days
customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0)
X = customers[['orders', 'days_since', 'tenure', 'avg_value']]
y = customers['next']
kf = KFold(n_splits=5, shuffle=True, random_state=42)

forest = RandomForestRegressor(n_estimators=50, max_depth=5, random_state=0)
forest_mae = ...
line_mae = ...
gap = ...
spread = ...
print(forest_mae.round(2), line_mae.round(2))
print(round(gap, 2), round(spread, 2))

from sklearn.ensemble import RandomForestRegressor import numpy as np import pandas as pd from sklearn.model_selection import KFold, cross_val_score, train_test_split from sklearn.linear_model import LinearRegression from sklearn.tree import DecisionTreeRegressor from sklearn.dummy import DummyRegressor from sklearn.metrics import mean_absolute_error orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') orders['ordered_at'] = pd.to_datetime(orders['ordered_at']) items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] customers = before.groupby('customer_id').agg( orders=('id', 'size'), last=('ordered_at', 'max'), first=('ordered_at', 'min'), avg_value=('value', 'mean'), ) customers['days_since'] = (cutoff - customers['last']).dt.days customers['tenure'] = (cutoff - customers['first']).dt.days customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0) X = customers[['orders', 'days_since', 'tenure', 'avg_value']] y = customers['next'] kf = KFold(n_splits=5, shuffle=True, random_state=42) forest = RandomForestRegressor(n_estimators=50, max_depth=5, random_state=0) forest_mae = ... line_mae = ... gap = ... spread = ... print(forest_mae.round(2), line_mae.round(2)) print(round(gap, 2), round(spread, 2))

Check▶ Run

The idea

Overfitting

Why does the tree have 3 levels and not 20? The deeper it is, the finer it splits the customers. Here is the error on the training customers and on the test ones for depths from 1 to 12:

Pythondata: the shop tables

import matplotlib.pyplot as plt

import numpy as np
import pandas as pd
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
orders['ordered_at'] = pd.to_datetime(orders['ordered_at'])
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]
customers = before.groupby('customer_id').agg(
    orders=('id', 'size'),
    last=('ordered_at', 'max'),
    first=('ordered_at', 'min'),
    avg_value=('value', 'mean'),
)
customers['days_since'] = (cutoff - customers['last']).dt.days
customers['tenure'] = (cutoff - customers['first']).dt.days
customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0)
X = customers[['orders', 'days_since', 'tenure', 'avg_value']]
y = customers['next']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

depths = range(1, 13)
train_mae = []
test_mae = []
for depth in depths:
    tree = DecisionTreeRegressor(max_depth=depth, random_state=0).fit(X_train, y_train)
    train_mae.append(mean_absolute_error(y_train, tree.predict(X_train)))
    test_mae.append(mean_absolute_error(y_test, tree.predict(X_test)))
print([round(m, 2) for m in train_mae])
print([round(m, 2) for m in test_mae])

deep = DecisionTreeRegressor(random_state=0).fit(X_train, y_train)
print(deep.get_depth(), round(mean_absolute_error(y_train, deep.predict(X_train)), 3), round(mean_absolute_error(y_test, deep.predict(X_test)), 2))

fig, ax = plt.subplots()
ax.plot(depths, train_mae, marker='o', label='training')
ax.plot(depths, test_mae, marker='o', label='test')
ax.set_title('Error of the tree by depth')
ax.set_xlabel('Depth (max_depth)')
ax.set_ylabel('MAE, orders per customer')
ax.set_ylim(0, None)
ax.legend()

import matplotlib.pyplot as plt import numpy as np import pandas as pd from sklearn.model_selection import KFold, cross_val_score, train_test_split from sklearn.linear_model import LinearRegression from sklearn.tree import DecisionTreeRegressor from sklearn.dummy import DummyRegressor from sklearn.metrics import mean_absolute_error orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') orders['ordered_at'] = pd.to_datetime(orders['ordered_at']) items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] customers = before.groupby('customer_id').agg( orders=('id', 'size'), last=('ordered_at', 'max'), first=('ordered_at', 'min'), avg_value=('value', 'mean'), ) customers['days_since'] = (cutoff - customers['last']).dt.days customers['tenure'] = (cutoff - customers['first']).dt.days customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0) X = customers[['orders', 'days_since', 'tenure', 'avg_value']] y = customers['next'] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) depths = range(1, 13) train_mae = [] test_mae = [] for depth in depths: tree = DecisionTreeRegressor(max_depth=depth, random_state=0).fit(X_train, y_train) train_mae.append(mean_absolute_error(y_train, tree.predict(X_train))) test_mae.append(mean_absolute_error(y_test, tree.predict(X_test))) print([round(m, 2) for m in train_mae]) print([round(m, 2) for m in test_mae]) deep = DecisionTreeRegressor(random_state=0).fit(X_train, y_train) print(deep.get_depth(), round(mean_absolute_error(y_train, deep.predict(X_train)), 3), round(mean_absolute_error(y_test, deep.predict(X_test)), 2)) fig, ax = plt.subplots() ax.plot(depths, train_mae, marker='o', label='training') ax.plot(depths, test_mae, marker='o', label='test') ax.set_title('Error of the tree by depth') ax.set_xlabel('Depth (max_depth)') ax.set_ylabel('MAE, orders per customer') ax.set_ylim(0, None) ax.legend()

▶ RunRestore the code

On the training customers the error falls every time: from 1.87 to 0.41. On the test customers it falls until depth 3 (1.50) and then grows to 1.91. A tree with no limit reaches 23 levels and is off by 0.001 on the training customers, almost zero, and by 2.00 on the test ones.

The deep tree has not learned more about the customers. It has memorized every training customer together with their noise: one customer happened to order more, another less. This is called overfitting. The sign is a big gap between the training error and the test error. That is why the error on the data the model learned from says nothing about new customers.

Try it

Pick the depth

Pick the depth of the tree with cross-validation. Store in cv_mae (a Series indexed by the depth from 1 to 10) the mean MAE from cross_val_score with the folds kf, and in best_depth the depth with the smallest error.

Pythondata: the shop tables

import numpy as np
import pandas as pd
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
orders['ordered_at'] = pd.to_datetime(orders['ordered_at'])
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]
customers = before.groupby('customer_id').agg(
    orders=('id', 'size'),
    last=('ordered_at', 'max'),
    first=('ordered_at', 'min'),
    avg_value=('value', 'mean'),
)
customers['days_since'] = (cutoff - customers['last']).dt.days
customers['tenure'] = (cutoff - customers['first']).dt.days
customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0)
X = customers[['orders', 'days_since', 'tenure', 'avg_value']]
y = customers['next']
kf = KFold(n_splits=5, shuffle=True, random_state=42)

cv_mae = pd.Series(dtype=float)
for depth in range(1, 11):
    tree = DecisionTreeRegressor(max_depth=depth, random_state=0)
    cv_mae[depth] = ...
best_depth = ...
print(cv_mae.round(2))
print(best_depth)

import numpy as np import pandas as pd from sklearn.model_selection import KFold, cross_val_score, train_test_split from sklearn.linear_model import LinearRegression from sklearn.tree import DecisionTreeRegressor from sklearn.dummy import DummyRegressor from sklearn.metrics import mean_absolute_error orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') orders['ordered_at'] = pd.to_datetime(orders['ordered_at']) items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] customers = before.groupby('customer_id').agg( orders=('id', 'size'), last=('ordered_at', 'max'), first=('ordered_at', 'min'), avg_value=('value', 'mean'), ) customers['days_since'] = (cutoff - customers['last']).dt.days customers['tenure'] = (cutoff - customers['first']).dt.days customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0) X = customers[['orders', 'days_since', 'tenure', 'avg_value']] y = customers['next'] kf = KFold(n_splits=5, shuffle=True, random_state=42) cv_mae = pd.Series(dtype=float) for depth in range(1, 11): tree = DecisionTreeRegressor(max_depth=depth, random_state=0) cv_mae[depth] = ... best_depth = ... print(cv_mae.round(2)) print(best_depth)

Check▶ Run

The idea

When the order is time

Customers can be shuffled: one customer knows nothing about another. Days cannot. If you predict the orders for tomorrow, the model will only have the past. Shuffled folds would also train it on days after the test days, and that is the leakage from the previous lesson.

TimeSeriesSplit splits by time: it learns on the start and checks on the following days, then learns on more and checks on even later ones. An example: the orders for a day from the day of the week, the orders 7 days earlier and the mean of the previous 7 days:

Pythondata: the shop tables

import numpy as np
import pandas as pd
from sklearn.model_selection import TimeSeriesSplit, cross_val_score
from sklearn.linear_model import LinearRegression
from sklearn.dummy import DummyRegressor

orders = pd.read_csv('year_orders.csv')
orders['ordered_at'] = pd.to_datetime(orders['ordered_at'])
daily = orders.set_index('ordered_at').resample('D').size().rename('orders').to_frame()
daily['weekday'] = daily.index.dayofweek
daily['last_week'] = daily['orders'].shift(7)
daily['week_mean'] = daily['orders'].shift(1).rolling(7).mean()
daily = daily.dropna()
X = pd.get_dummies(daily['weekday'], prefix='day', dtype=int).join(daily[['last_week', 'week_mean']])
y = daily['orders']

print(len(daily), daily.index[0].date(), daily.index[-1].date())
ts = TimeSeriesSplit(n_splits=5)
for train, test in ts.split(X):
    print(daily.index[train[0]].date(), '…', daily.index[train[-1]].date(), '→', daily.index[test[0]].date(), '…', daily.index[test[-1]].date())

mae = -cross_val_score(LinearRegression(), X, y, cv=ts, scoring='neg_mean_absolute_error')
print(mae.round(2), f'{mae.mean():.2f} ± {mae.std():.2f}')

import numpy as np import pandas as pd from sklearn.model_selection import TimeSeriesSplit, cross_val_score from sklearn.linear_model import LinearRegression from sklearn.dummy import DummyRegressor orders = pd.read_csv('year_orders.csv') orders['ordered_at'] = pd.to_datetime(orders['ordered_at']) daily = orders.set_index('ordered_at').resample('D').size().rename('orders').to_frame() daily['weekday'] = daily.index.dayofweek daily['last_week'] = daily['orders'].shift(7) daily['week_mean'] = daily['orders'].shift(1).rolling(7).mean() daily = daily.dropna() X = pd.get_dummies(daily['weekday'], prefix='day', dtype=int).join(daily[['last_week', 'week_mean']]) y = daily['orders'] print(len(daily), daily.index[0].date(), daily.index[-1].date()) ts = TimeSeriesSplit(n_splits=5) for train, test in ts.split(X): print(daily.index[train[0]].date(), '…', daily.index[train[-1]].date(), '→', daily.index[test[0]].date(), '…', daily.index[test[-1]].date()) mae = -cross_val_score(LinearRegression(), X, y, cv=ts, scoring='neg_mean_absolute_error') print(mae.round(2), f'{mae.mean():.2f} ± {mae.std():.2f}')

▶ RunRestore the code

The first 7 days have no “7 days earlier” and drop out, 358 remain. Each test fold is 59 days after the training part, and the training part grows: 2 months at first, 10 at the end. The line is off by 4.51 ± 0.18 orders per day on average.

Try it

Does it beat the baseline?

Compare the line with the baseline (DummyRegressor, the mean of the training days) using TimeSeriesSplit with 5 folds. Store in line_mae and base_mae the five errors (MAE, positive) of each.

Pythondata: the shop tables

import numpy as np
import pandas as pd
from sklearn.model_selection import TimeSeriesSplit, cross_val_score
from sklearn.linear_model import LinearRegression
from sklearn.dummy import DummyRegressor

orders = pd.read_csv('year_orders.csv')
orders['ordered_at'] = pd.to_datetime(orders['ordered_at'])
daily = orders.set_index('ordered_at').resample('D').size().rename('orders').to_frame()
daily['weekday'] = daily.index.dayofweek
daily['last_week'] = daily['orders'].shift(7)
daily['week_mean'] = daily['orders'].shift(1).rolling(7).mean()
daily = daily.dropna()
X = pd.get_dummies(daily['weekday'], prefix='day', dtype=int).join(daily[['last_week', 'week_mean']])
y = daily['orders']

ts = TimeSeriesSplit(n_splits=5)
line_mae = ...
base_mae = ...
print(line_mae.round(2), base_mae.round(2))
print(f'{line_mae.mean():.2f} ± {line_mae.std():.2f}', f'{base_mae.mean():.2f} ± {base_mae.std():.2f}')

import numpy as np import pandas as pd from sklearn.model_selection import TimeSeriesSplit, cross_val_score from sklearn.linear_model import LinearRegression from sklearn.dummy import DummyRegressor orders = pd.read_csv('year_orders.csv') orders['ordered_at'] = pd.to_datetime(orders['ordered_at']) daily = orders.set_index('ordered_at').resample('D').size().rename('orders').to_frame() daily['weekday'] = daily.index.dayofweek daily['last_week'] = daily['orders'].shift(7) daily['week_mean'] = daily['orders'].shift(1).rolling(7).mean() daily = daily.dropna() X = pd.get_dummies(daily['weekday'], prefix='day', dtype=int).join(daily[['last_week', 'week_mean']]) y = daily['orders'] ts = TimeSeriesSplit(n_splits=5) line_mae = ... base_mae = ... print(line_mae.round(2), base_mae.round(2)) print(f'{line_mae.mean():.2f} ± {line_mae.std():.2f}', f'{base_mae.mean():.2f} ± {base_mae.std():.2f}')

Check▶ Run

The idea

The test is touched once

If you try 10 models on the test part and take the best, the test took part in the choice. One of the ten will be lucky with exactly these customers, and its error will look better than it is.

So the order is:

1. You set aside a test part at the start and do not look at it.
2. You choose the model and its settings (depth, features) with cross-validation on the training part only.
3. You train the chosen model on the whole training part and measure it on the test once. That is the number you tell the manager.

Pythondata: the shop tables

import numpy as np
import pandas as pd
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
orders['ordered_at'] = pd.to_datetime(orders['ordered_at'])
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]
customers = before.groupby('customer_id').agg(
    orders=('id', 'size'),
    last=('ordered_at', 'max'),
    first=('ordered_at', 'min'),
    avg_value=('value', 'mean'),
)
customers['days_since'] = (cutoff - customers['last']).dt.days
customers['tenure'] = (cutoff - customers['first']).dt.days
customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0)
X = customers[['orders', 'days_since', 'tenure', 'avg_value']]
y = customers['next']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
kf = KFold(n_splits=5, shuffle=True, random_state=42)
print(len(X_train), len(X_test))

models = [
    ('mean', DummyRegressor()),
    ('line', LinearRegression()),
    ('tree', DecisionTreeRegressor(max_depth=3, random_state=0)),
]
for name, model in models:
    mae = -cross_val_score(model, X_train, y_train, cv=kf, scoring='neg_mean_absolute_error')
    print(name, f'{mae.mean():.2f} ± {mae.std():.2f}')

import numpy as np import pandas as pd from sklearn.model_selection import KFold, cross_val_score, train_test_split from sklearn.linear_model import LinearRegression from sklearn.tree import DecisionTreeRegressor from sklearn.dummy import DummyRegressor from sklearn.metrics import mean_absolute_error orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') orders['ordered_at'] = pd.to_datetime(orders['ordered_at']) items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] customers = before.groupby('customer_id').agg( orders=('id', 'size'), last=('ordered_at', 'max'), first=('ordered_at', 'min'), avg_value=('value', 'mean'), ) customers['days_since'] = (cutoff - customers['last']).dt.days customers['tenure'] = (cutoff - customers['first']).dt.days customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0) X = customers[['orders', 'days_since', 'tenure', 'avg_value']] y = customers['next'] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) kf = KFold(n_splits=5, shuffle=True, random_state=42) print(len(X_train), len(X_test)) models = [ ('mean', DummyRegressor()), ('line', LinearRegression()), ('tree', DecisionTreeRegressor(max_depth=3, random_state=0)), ] for name, model in models: mae = -cross_val_score(model, X_train, y_train, cv=kf, scoring='neg_mean_absolute_error') print(name, f'{mae.mean():.2f} ± {mae.std():.2f}')

▶ RunRestore the code

On the 1,023 training customers the baseline is off by 2.74 ± 0.17, the line by 1.32 ± 0.06, and the tree by 1.36 ± 0.07. The line has the smaller error and is simpler: you choose it. The 256 test customers are still untouched.

More precisely, if you are curious

If after the test you change the model and measure on the test again, the test becomes part of the choice. Then you need a new test part: new data, for example the next month. That is why for a model of orders the most honest check is the time after the training, as in TimeSeriesSplit.

The idea

When a model is not needed

A model costs time: data, training, watching that it does not go bad. Sometimes it is not worth it:

• Little data. With a few dozen customers the cross-validation error jumps so much that you cannot say whether the model beats the mean.
• A rule without a model does the same. “The customer will buy at the same pace” is one division. If the model does not beat it clearly, the rule is better: everyone understands it and nothing can break.
• The cost of an error. If an error is expensive (you refuse a loan, block a card), the model must be very sure, or a person must check.
• Nobody acts on the prediction. If no decision follows from the number (who gets a coupon, how much stock), the model is only an exercise.

The line has a coefficient of 0.469 for the number of orders so far, and the other features barely take part. So it is almost “half of the orders so far”: 8 months before, 4 months after. Check it in the exercise.

Exercise

The verdict

Cross-validation on the training part chose the line. Train it on X_train and check it once on the test: store in model the trained model, in test_mae its error (MAE) on the test customers, and in base_mae the error of the baseline on the test.

Pythondata: the shop tables

import numpy as np
import pandas as pd
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
orders['ordered_at'] = pd.to_datetime(orders['ordered_at'])
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]
customers = before.groupby('customer_id').agg(
    orders=('id', 'size'),
    last=('ordered_at', 'max'),
    first=('ordered_at', 'min'),
    avg_value=('value', 'mean'),
)
customers['days_since'] = (cutoff - customers['last']).dt.days
customers['tenure'] = (cutoff - customers['first']).dt.days
customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0)
X = customers[['orders', 'days_since', 'tenure', 'avg_value']]
y = customers['next']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = ...
test_mae = ...
base = DummyRegressor().fit(X_train, y_train)
base_mae = ...
print(round(test_mae, 2), round(base_mae, 2))

import numpy as np import pandas as pd from sklearn.model_selection import KFold, cross_val_score, train_test_split from sklearn.linear_model import LinearRegression from sklearn.tree import DecisionTreeRegressor from sklearn.dummy import DummyRegressor from sklearn.metrics import mean_absolute_error orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') orders['ordered_at'] = pd.to_datetime(orders['ordered_at']) items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] customers = before.groupby('customer_id').agg( orders=('id', 'size'), last=('ordered_at', 'max'), first=('ordered_at', 'min'), avg_value=('value', 'mean'), ) customers['days_since'] = (cutoff - customers['last']).dt.days customers['tenure'] = (cutoff - customers['first']).dt.days customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0) X = customers[['orders', 'days_since', 'tenure', 'avg_value']] y = customers['next'] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) model = ... test_mae = ... base = DummyRegressor().fit(X_train, y_train) base_mae = ... print(round(test_mae, 2), round(base_mae, 2))

Check▶ Run

Does the line beat a rule with no model? The rule is “the customer will buy at the same pace”: for 4 months, half of their orders over the 8 months before. Store in rule the rule's prediction for each test customer, and in rule_mae its error on the test.

Pythondata: the shop tables

import numpy as np
import pandas as pd
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error

orders = pd.read_csv('year_orders.csv')
items = pd.read_csv('year_items.csv')
orders['ordered_at'] = pd.to_datetime(orders['ordered_at'])
items['revenue'] = items['quantity'] * items['unit_price']
orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]
customers = before.groupby('customer_id').agg(
    orders=('id', 'size'),
    last=('ordered_at', 'max'),
    first=('ordered_at', 'min'),
    avg_value=('value', 'mean'),
)
customers['days_since'] = (cutoff - customers['last']).dt.days
customers['tenure'] = (cutoff - customers['first']).dt.days
customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0)
X = customers[['orders', 'days_since', 'tenure', 'avg_value']]
y = customers['next']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LinearRegression().fit(X_train, y_train)
model_mae = mean_absolute_error(y_test, model.predict(X_test))
rule = ...
rule_mae = ...
print(round(model_mae, 2), round(rule_mae, 2))
print(dict(zip(X.columns, model.coef_.round(3).tolist())))

import numpy as np import pandas as pd from sklearn.model_selection import KFold, cross_val_score, train_test_split from sklearn.linear_model import LinearRegression from sklearn.tree import DecisionTreeRegressor from sklearn.dummy import DummyRegressor from sklearn.metrics import mean_absolute_error orders = pd.read_csv('year_orders.csv') items = pd.read_csv('year_items.csv') orders['ordered_at'] = pd.to_datetime(orders['ordered_at']) items['revenue'] = items['quantity'] * items['unit_price'] orders['value'] = orders['id'].map(items.groupby('order_id')['revenue'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] customers = before.groupby('customer_id').agg( orders=('id', 'size'), last=('ordered_at', 'max'), first=('ordered_at', 'min'), avg_value=('value', 'mean'), ) customers['days_since'] = (cutoff - customers['last']).dt.days customers['tenure'] = (cutoff - customers['first']).dt.days customers['next'] = after.groupby('customer_id').size().reindex(customers.index, fill_value=0) X = customers[['orders', 'days_since', 'tenure', 'avg_value']] y = customers['next'] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) model = LinearRegression().fit(X_train, y_train) model_mae = mean_absolute_error(y_test, model.predict(X_test)) rule = ... rule_mae = ... print(round(model_mae, 2), round(rule_mae, 2)) print(dict(zip(X.columns, model.coef_.round(3).tolist())))

Check▶ Run

Test

Check yourself

1. Why does cross-validation give a more reliable score than one split?

Because the model learns on more data and gets more accurateEvery customer is in the test exactly once, the score is a mean of 5 checks, and you see how much they jumpBecause it uses no test part

2. A tree with no limit is off by 0.001 orders on the training customers and by 2.00 on the test ones. What does that mean?

The tree is almost perfectThe tree has memorized the training customers together with their noise: that is overfittingThe test data is wrong

3. You predict the orders for a day. Why TimeSeriesSplit and not KFold(shuffle=True)?

TimeSeriesSplit is fasterThe model must always learn on the past and be checked on later days, the way it will be usedKFold does not work with dates

4. You checked 10 models on the test part and picked the best. How honest is its test error?

Too pessimisticFully honest, it is the test after allToo optimistic: the test took part in the choice, so it is no longer new

5. The rule “half of the orders so far” is off by 1.41, and the line by 1.38. What is sensible?

Look for a more complex model until it beats the rule by a lotUse the rule: it is almost as accurate, everyone understands it and there is nothing to maintainUse the line, because it is 0.03 more accurate

Check

Start the lesson over

Related lessons

• Builds on: Models: learning from examples · Linear regression · Classification · Data leakage
• Tasks on this topic: Models with scikit-learn