Data leakage

In short: Data leakage is when a model sees information during training that it will not have at real prediction time, so the test score looks better than the model will be in real use.

Problem

99.7% accuracy

A colleague has built the model from the classification lesson: which customers will not order after 1 July. Only they take the customer table straight from the system, as it is today, 1 November: number of orders, days since the last order and average order. And they show you:

Pythondata: the shop tables

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['value'] = items['quantity'] * items['unit_price']
orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]

today = pd.Timestamp('2026-11-01')
people = orders.groupby('customer_id').agg(
    orders=('id', 'size'), first=('ordered_at', 'min'),
    last=('ordered_at', 'max'), basket=('basket', 'mean'))
people['days_since'] = (today - people['last']).dt.days
people = people[people['first'] < cutoff]
people['lost'] = (~people.index.isin(after['customer_id'])).astype(int)

X = people[['orders', 'days_since', 'basket']]
y = people['lost']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)
model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
print('customers:', len(people))
print('baseline:', round(1 - y_test.mean(), 3))
print('accuracy:', round(model.score(X_test, y_test), 3))

import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['value'] = items['quantity'] * items['unit_price'] orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] today = pd.Timestamp('2026-11-01') people = orders.groupby('customer_id').agg( orders=('id', 'size'), first=('ordered_at', 'min'), last=('ordered_at', 'max'), basket=('basket', 'mean')) people['days_since'] = (today - people['last']).dt.days people = people[people['first'] < cutoff] people['lost'] = (~people.index.isin(after['customer_id'])).astype(int) X = people[['orders', 'days_since', 'basket']] y = people['lost'] X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, random_state=42, stratify=y) model = LogisticRegression(max_iter=1000).fit(X_train, y_train) print('customers:', len(people)) print('baseline:', round(1 - y_test.mean(), 3)) print('accuracy:', round(model.score(X_test, y_test), 3))

▶ RunRestore the code

There are 1,279 customers. The rule “everyone comes back” gets 72.5% of the test customers right. The model gets 99.7%: out of 320 customers in the test set it misses one. In the classification lesson, with the same three features, the model had 72.2%, below the rule. What changed? The table.

The idea

The answer is in the features

See how many days since the last order the returning customers (lost is 0) and the lost ones (lost is 1) have:

Pythondata: the shop tables

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['value'] = items['quantity'] * items['unit_price']
orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]

today = pd.Timestamp('2026-11-01')
people = orders.groupby('customer_id').agg(
    orders=('id', 'size'), first=('ordered_at', 'min'),
    last=('ordered_at', 'max'), basket=('basket', 'mean'))
people['days_since'] = (today - people['last']).dt.days
people = people[people['first'] < cutoff]
people['lost'] = (~people.index.isin(after['customer_id'])).astype(int)

people.groupby('lost')['days_since'].agg(['min', 'max'])

import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['value'] = items['quantity'] * items['unit_price'] orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] today = pd.Timestamp('2026-11-01') people = orders.groupby('customer_id').agg( orders=('id', 'size'), first=('ordered_at', 'min'), last=('ordered_at', 'max'), basket=('basket', 'mean')) people['days_since'] = (today - people['last']).dt.days people = people[people['first'] < cutoff] people['lost'] = (~people.index.isin(after['customer_id'])).astype(int) people.groupby('lost')['days_since'].agg(['min', 'max'])

▶ RunRestore the code

Every returning customer has from 0 to 122 days, and every lost one from 123 to 363. One number splits them without a mistake. Why? days_since is computed up to today. Whoever ordered after 1 July has a last order after 1 July, so they are less than 123 days away from 1 November. The feature contains the label. The same goes for orders: it counts the orders after 1 July too.

This is called information leakage (data leakage): information that the model will not have when it predicts gets into the training. On 1 July you do not know who will order in August. The rule is one: every feature is computed only from what is known at the moment of the prediction.

More precisely, if you are curious

Leakage hides in columns that the system updates after the event: “total orders”, “last purchase”, “status”, “returned item”. For every column ask when its value is written. If it is after the day of the prediction, the column is no good as a feature, however useful it looks. The strongest features in a model are also the first ones you check.

The idea

The honest number

The same features twice. Once from all the orders up to today, once only from the orders before 1 July and up to 1 July:

Pythondata: the shop tables

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['value'] = items['quantity'] * items['unit_price']
orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]

def features(rows, now):
    people = rows.groupby('customer_id').agg(
        orders=('id', 'size'), last=('ordered_at', 'max'), basket=('basket', 'mean'))
    people['days_since'] = (now - people['last']).dt.days
    return people[['orders', 'days_since', 'basket']]

honest = features(before, cutoff)
leaky = features(orders, pd.Timestamp('2026-11-01')).loc[honest.index]
y = (~honest.index.isin(after['customer_id'])).astype(int)

for name, X in [('with the future', leaky), ('honest', honest)]:
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.25, random_state=42, stratify=y)
    model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
    print(name, round(model.score(X_test, y_test), 3))
print('customers:', len(honest), 'train:', len(X_train), 'test:', len(X_test))
print('baseline:', round(1 - y_test.mean(), 3))

import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['value'] = items['quantity'] * items['unit_price'] orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] def features(rows, now): people = rows.groupby('customer_id').agg( orders=('id', 'size'), last=('ordered_at', 'max'), basket=('basket', 'mean')) people['days_since'] = (now - people['last']).dt.days return people[['orders', 'days_since', 'basket']] honest = features(before, cutoff) leaky = features(orders, pd.Timestamp('2026-11-01')).loc[honest.index] y = (~honest.index.isin(after['customer_id'])).astype(int) for name, X in [('with the future', leaky), ('honest', honest)]: X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, random_state=42, stratify=y) model = LogisticRegression(max_iter=1000).fit(X_train, y_train) print(name, round(model.score(X_test, y_test), 3)) print('customers:', len(honest), 'train:', len(X_train), 'test:', len(X_test)) print('baseline:', round(1 - y_test.mean(), 3))

▶ RunRestore the code

With the future in the features the accuracy is 99.7%. Honestly it is 72.2%, and the baseline is 72.5%. With these three features the model gains nothing in accuracy, just like in the classification lesson. Unpleasant, but true. Better to know it now than after you have handed out the coupons.

features(rows, now) is one function for both tables: the only difference is which orders you give it and up to which day it counts. .loc[honest.index] takes the same 1,279 customers in the same order, so the two splits are identical: 959 customers for training and 320 for the test.

See it

How the future gets into the feature

The same code, slowly. The dots are the orders of four customers by month, and the wall is 1 July. The label comes from the dots to the right of it. The colleague's feature counts them too.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['value'] = items['quantity'] * items['unit_price']
orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]

def features(rows, now):
    people = rows.groupby('customer_id').agg(
        orders=('id', 'size'), last=('ordered_at', 'max'), basket=('basket', 'mean'))
    people['days_since'] = (now - people['last']).dt.days
    return people[['orders', 'days_since', 'basket']]

honest = features(before, cutoff)
leaky = features(orders, pd.Timestamp('2026-11-01')).loc[honest.index]
y = (~honest.index.isin(after['customer_id'])).astype(int)

for name, X in [('with the future', leaky), ('honest', honest)]:
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.25, random_state=42, stratify=y)
    model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
    print(name, round(model.score(X_test, y_test), 3))
print('customers:', len(honest), 'train:', len(X_train), 'test:', len(X_test))
print('baseline:', round(1 - y_test.mean(), 3))

Step by step, in words

1. orders is every order of the year, up to 31 October 2026. The dots are the orders of four customers by month.
2. cutoff is 1 July. Left of the wall is before: what you know. Right of it is after: what you predict.
3. lost is 1 if the customer has no order at all on the right. The label is made from the dots after 1 July.
4. The colleague's table counts up to today. orders counts the dots on the right too, and days_since is measured from today: the same dots the label is made from.
5. The split: 959 customers for training and 320 for the test, out of 1,279.
6. With the future in the features the accuracy is 99.7%. The baseline “everyone comes back” is 72.5%.
7. Honestly: the features stop at 1 July. The dots on the right are not counted, and days_since is measured from the wall.
8. The honest accuracy is 72.2%, below the baseline. 99.7% was leakage, not a model.

The customers in the scene are a sketch: the numbers in the captions are real, from the number of customers to the two accuracies.

Try it

One more feature

1. On 1 July you decide who gets a coupon. Which feature is honest for the model?

The number of orders of the customer as it is in the system today, 1 NovemberThe number of orders of the customer up to 30 JuneThe days from the last order to today

Check

The colleague wants one more feature: recent, the number of orders of the customer in the last 30 days. Compute it honestly, as of 1 July: in month the orders of the 30 days before cutoff, in people['recent'] their number for each customer (0 for those without an order), and in score the accuracy of the model with the four features.

Pythondata: the shop tables

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['value'] = items['quantity'] * items['unit_price']
orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]

people = before.groupby('customer_id').agg(
    orders=('id', 'size'), last=('ordered_at', 'max'), basket=('basket', 'mean'))
people['days_since'] = (cutoff - people['last']).dt.days
people['lost'] = (~people.index.isin(after['customer_id'])).astype(int)

month = ...
people['recent'] = ...

X = people[['orders', 'days_since', 'basket', 'recent']]
y = people['lost']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)
model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
score = ...
print(round(people['recent'].mean(), 3), round(score, 3))

import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['value'] = items['quantity'] * items['unit_price'] orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] people = before.groupby('customer_id').agg( orders=('id', 'size'), last=('ordered_at', 'max'), basket=('basket', 'mean')) people['days_since'] = (cutoff - people['last']).dt.days people['lost'] = (~people.index.isin(after['customer_id'])).astype(int) month = ... people['recent'] = ... X = people[['orders', 'days_since', 'basket', 'recent']] y = people['lost'] X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, random_state=42, stratify=y) model = LogisticRegression(max_iter=1000).fit(X_train, y_train) score = ... print(round(people['recent'].mean(), 3), round(score, 3))

Check▶ Run

The idea

Preparation before the split

Logistic regression works better when the features are on a similar scale: days_since goes up to 240, and orders rarely goes over 10. StandardScaler subtracts from every feature its mean and divides by the standard deviation. That way every feature becomes centred on 0, with a spread of 1. The scaler learns the means with fit, from the data you give it. The colleague gives it everything:

Pythondata: the shop tables

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['value'] = items['quantity'] * items['unit_price']
orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]
from sklearn.preprocessing import StandardScaler

people = before.groupby('customer_id').agg(
    orders=('id', 'size'), last=('ordered_at', 'max'), basket=('basket', 'mean'))
people['days_since'] = (cutoff - people['last']).dt.days
people['lost'] = (~people.index.isin(after['customer_id'])).astype(int)

X = people[['orders', 'days_since', 'basket']]
y = people['lost']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

scaler = StandardScaler().fit(X)
print(scaler.mean_.round(2))
print(X_train.mean().round(2).to_numpy())
model = LogisticRegression(max_iter=1000).fit(scaler.transform(X_train), y_train)
print('accuracy:', round(model.score(scaler.transform(X_test), y_test), 3))

import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['value'] = items['quantity'] * items['unit_price'] orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] from sklearn.preprocessing import StandardScaler people = before.groupby('customer_id').agg( orders=('id', 'size'), last=('ordered_at', 'max'), basket=('basket', 'mean')) people['days_since'] = (cutoff - people['last']).dt.days people['lost'] = (~people.index.isin(after['customer_id'])).astype(int) X = people[['orders', 'days_since', 'basket']] y = people['lost'] X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, random_state=42, stratify=y) scaler = StandardScaler().fit(X) print(scaler.mean_.round(2)) print(X_train.mean().round(2).to_numpy()) model = LogisticRegression(max_iter=1000).fit(scaler.transform(X_train), y_train) print('accuracy:', round(model.score(scaler.transform(X_test), y_test), 3))

▶ RunRestore the code

The scaler learned from all 1,279 customers: the means are 5.88 orders, 67.42 days and €10.69. The training set alone has 5.96, 67.84 and 10.68. The difference comes from the 320 test customers: they affected the preparation before the model saw them. The test set is no longer quite unseen.

Here the accuracy does not change: it is 72.2% again, because the means are close. It is not always like that. Leakage grows with little data and with preparation that looks at the label, for example when you choose the features by how they relate to lost in all the rows. Filling missing values with the mean (fillna) is the same case: the mean is computed from the training set only.

The idea

Pipeline: the preparation learns from the training set only

A Pipeline joins the preparation and the model into one object. make_pipeline(StandardScaler(), LogisticRegression()) makes a chain of two steps:

• pipe.fit(X_train, y_train) trains the scaler and then the model, on the training set only;
• pipe.score(X_test, y_test) and pipe.predict scale the test set with the learned means and ask the model.

Pythondata: the shop tables

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['value'] = items['quantity'] * items['unit_price']
orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

people = before.groupby('customer_id').agg(
    orders=('id', 'size'), last=('ordered_at', 'max'), basket=('basket', 'mean'))
people['days_since'] = (cutoff - people['last']).dt.days
people['lost'] = (~people.index.isin(after['customer_id'])).astype(int)

X = people[['orders', 'days_since', 'basket']]
y = people['lost']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

pipe = make_pipeline(StandardScaler(), LogisticRegression())
pipe.fit(X_train, y_train)
print(pipe[0].mean_.round(2))
print('accuracy:', round(pipe.score(X_test, y_test), 3))

import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['value'] = items['quantity'] * items['unit_price'] orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] from sklearn.preprocessing import StandardScaler from sklearn.pipeline import make_pipeline people = before.groupby('customer_id').agg( orders=('id', 'size'), last=('ordered_at', 'max'), basket=('basket', 'mean')) people['days_since'] = (cutoff - people['last']).dt.days people['lost'] = (~people.index.isin(after['customer_id'])).astype(int) X = people[['orders', 'days_since', 'basket']] y = people['lost'] X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, random_state=42, stratify=y) pipe = make_pipeline(StandardScaler(), LogisticRegression()) pipe.fit(X_train, y_train) print(pipe[0].mean_.round(2)) print('accuracy:', round(pipe.score(X_test, y_test), 3))

▶ RunRestore the code

The means of the scaler in pipe are 5.96, 67.84 and 10.68: exactly those of the training set. The accuracy is 72.2%. Now the order cannot go wrong: the preparation is part of the model. A Pipeline helps in the next lesson too: in cross-validation every split learns the scaler again.

1. What does pipe.score(X_test, y_test) do when pipe = make_pipeline(StandardScaler(), LogisticRegression()) was trained on the training set?

It scales the test set with the means from the training set and asks the modelIt learns the scaler again from the test set and then asks the modelIt learns both the scaler and the model again from the test set

Check

More precisely, if you are curious

pipe[0] is the first step, and pipe[-1] is the model. A chain can have more steps: SimpleImputer fills the missing values, OneHotEncoder makes 0/1 columns from categories, and ColumnTransformer applies different preparation to different columns. All of them learn with one fit, from the training set only.

The idea

One customer in both training and test

The table was glued together from two exports, and every customer is in it twice. In the code pd.concat does that: it puts the two tables one under the other. It looks harmless: the same data, just more of it. You try a decision tree (DecisionTreeClassifier) that grows without a depth limit:

Pythondata: the shop tables

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['value'] = items['quantity'] * items['unit_price']
orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]
from sklearn.tree import DecisionTreeClassifier

people = before.groupby('customer_id').agg(
    orders=('id', 'size'), last=('ordered_at', 'max'), basket=('basket', 'mean'))
people['days_since'] = (cutoff - people['last']).dt.days
people['lost'] = (~people.index.isin(after['customer_id'])).astype(int)

rows = pd.concat([people, people]).reset_index()
print(len(rows), rows.duplicated().sum())

for name, data in [('with duplicates', rows), ('without duplicates', rows.drop_duplicates())]:
    X = data[['orders', 'days_since', 'basket']]
    y = data['lost']
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.25, random_state=42, stratify=y)
    tree = DecisionTreeClassifier(random_state=42).fit(X_train, y_train)
    print(name, len(data), round(tree.score(X_test, y_test), 3))

import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['value'] = items['quantity'] * items['unit_price'] orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] from sklearn.tree import DecisionTreeClassifier people = before.groupby('customer_id').agg( orders=('id', 'size'), last=('ordered_at', 'max'), basket=('basket', 'mean')) people['days_since'] = (cutoff - people['last']).dt.days people['lost'] = (~people.index.isin(after['customer_id'])).astype(int) rows = pd.concat([people, people]).reset_index() print(len(rows), rows.duplicated().sum()) for name, data in [('with duplicates', rows), ('without duplicates', rows.drop_duplicates())]: X = data[['orders', 'days_since', 'basket']] y = data['lost'] X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, random_state=42, stratify=y) tree = DecisionTreeClassifier(random_state=42).fit(X_train, y_train) print(name, len(data), round(tree.score(X_test, y_test), 3))

▶ RunRestore the code

There are 2,558 rows, and 1,279 of them are repeats. With the duplicates the tree gets 89.4% of the test set right, above the baseline. Without them it gets 58.4%. With a random split most test rows have a twin in the training set, and a tree without a limit memorises every row. It does not predict, it remembers. A new customer has no twin: 58.4% is the real number, below the baseline, because the tree is overfitted. That is what the next lesson is about.

The rule: you remove the duplicates (drop_duplicates) before the split. Here reset_index() makes customer_id a column, so that the repeats of the same customer are removed, and not two different customers with the same numbers.

More precisely, if you are curious

Sometimes a customer has several real rows, for example one for every order. There are no duplicates, but if their rows are in both the training and the test set, the model again learns the customer, not the rule. Then you split by customer: GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=42).split(X, y, groups=rows['customer_id']) puts all the rows of one customer on one side.

The idea

Random days from a time series

In the regression lesson you split by time. Here is why a random split lies. The prediction is for the orders per day, from the number of the day and the day of the week. The model is a random forest (RandomForestRegressor): many trees, each trained on slightly different rows, and the prediction is their average.

Pythondata: the shop tables

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
daily = orders.set_index('ordered_at').resample('D').size().rename('orders').to_frame()
daily['day'] = range(len(daily))
daily['weekday'] = daily.index.dayofweek
X = daily[['day', 'weekday']]
y = daily['orders']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
forest = RandomForestRegressor(n_estimators=50, random_state=42).fit(X_train, y_train)
base = DummyRegressor().fit(X_train, y_train)
print('random days:', round(mean_absolute_error(y_test, forest.predict(X_test)), 2),
      'mean:', round(mean_absolute_error(y_test, base.predict(X_test)), 2))

late = daily.index >= '2026-08-01'
forest = RandomForestRegressor(n_estimators=50, random_state=42).fit(X[~late], y[~late])
base = DummyRegressor().fit(X[~late], y[~late])
print('August-October:', round(mean_absolute_error(y[late], forest.predict(X[late])), 2),
      'mean:', round(mean_absolute_error(y[late], base.predict(X[late])), 2))

import pandas as pd from sklearn.model_selection import train_test_split from sklearn.ensemble import RandomForestRegressor from sklearn.dummy import DummyRegressor from sklearn.metrics import mean_absolute_error orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) daily = orders.set_index('ordered_at').resample('D').size().rename('orders').to_frame() daily['day'] = range(len(daily)) daily['weekday'] = daily.index.dayofweek X = daily[['day', 'weekday']] y = daily['orders'] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42) forest = RandomForestRegressor(n_estimators=50, random_state=42).fit(X_train, y_train) base = DummyRegressor().fit(X_train, y_train) print('random days:', round(mean_absolute_error(y_test, forest.predict(X_test)), 2), 'mean:', round(mean_absolute_error(y_test, base.predict(X_test)), 2)) late = daily.index >= '2026-08-01' forest = RandomForestRegressor(n_estimators=50, random_state=42).fit(X[~late], y[~late]) base = DummyRegressor().fit(X[~late], y[~late]) print('August-October:', round(mean_absolute_error(y[late], forest.predict(X[late])), 2), 'mean:', round(mean_absolute_error(y[late], base.predict(X[late])), 2))

▶ RunRestore the code

With random days the forest is off by 4.88 orders a day on average, and the mean (DummyRegressor) by 5.76: the forest looks better. Trained up to 31 July and tried on August to October, the forest is off by 7.96, and the mean by 5.44. The forest is worse than the mean.

With a random split, to get 15 September right the forest has seen 14 and 16 September. In real life on 31 July you do not have a single day of September. Besides, trees do not continue a trend: a day after the last known one is, for them, like the last known days. And August to October has a surprise too: September is weaker. Why, we will find out in the project. A model trained up to July cannot know it, and neither can you on 31 July. That is why 7.96 is the honest number.

Exercise

Fix the colleague's model

Fix the colleague's code. The features come only from the orders before cutoff, and days_since is counted up to cutoff. The scaler and the logistic regression are in pipe, trained on the training set only. Store in score the accuracy of pipe on the test set.

Pythondata: the shop tables

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['value'] = items['quantity'] * items['unit_price']
orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum())

cutoff = pd.Timestamp('2026-07-01')
before = orders[orders['ordered_at'] < cutoff]
after = orders[orders['ordered_at'] >= cutoff]
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

today = pd.Timestamp('2026-11-01')
people = orders.groupby('customer_id').agg(
    orders=('id', 'size'), first=('ordered_at', 'min'),
    last=('ordered_at', 'max'), basket=('basket', 'mean'))
people['days_since'] = (today - people['last']).dt.days
people = people[people['first'] < cutoff]
people['lost'] = (~people.index.isin(after['customer_id'])).astype(int)

X = people[['orders', 'days_since', 'basket']]
y = people['lost']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

pipe = ...
...
score = ...
print(round(score, 3))

import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['value'] = items['quantity'] * items['unit_price'] orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum()) cutoff = pd.Timestamp('2026-07-01') before = orders[orders['ordered_at'] < cutoff] after = orders[orders['ordered_at'] >= cutoff] from sklearn.preprocessing import StandardScaler from sklearn.pipeline import make_pipeline today = pd.Timestamp('2026-11-01') people = orders.groupby('customer_id').agg( orders=('id', 'size'), first=('ordered_at', 'min'), last=('ordered_at', 'max'), basket=('basket', 'mean')) people['days_since'] = (today - people['last']).dt.days people = people[people['first'] < cutoff] people['lost'] = (~people.index.isin(after['customer_id'])).astype(int) X = people[['orders', 'days_since', 'basket']] y = people['lost'] X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, random_state=42, stratify=y) pipe = ... ... score = ... print(round(score, 3))

Check▶ Run

The colleague also has a model for the revenue per day, without the cancelled orders. Try it honestly. Train the random forest (n_estimators=50, random_state=42) and a DummyRegressor on the days up to 31 July (late is the mask of the days from 1 August). Then store in mae the error (MAE) of the forest on August to October, and in base_mae the error of the mean.

Pythondata: the shop tables

import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error

orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at'])
items = pd.read_csv('year_items.csv')
items['value'] = items['quantity'] * items['unit_price']
orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum())
orders = orders[orders['status'] != 'cancelled']
daily = orders.set_index('ordered_at').resample('D')['basket'].sum().rename('revenue').to_frame()
daily['day'] = range(len(daily))
daily['weekday'] = daily.index.dayofweek
X = daily[['day', 'weekday']]
y = daily['revenue']

late = ...
forest = RandomForestRegressor(n_estimators=50, random_state=42).fit(...)
base = DummyRegressor().fit(...)
mae = ...
base_mae = ...
print(round(mae, 2), round(base_mae, 2))

import pandas as pd from sklearn.ensemble import RandomForestRegressor from sklearn.dummy import DummyRegressor from sklearn.metrics import mean_absolute_error orders = pd.read_csv('year_orders.csv', parse_dates=['ordered_at']) items = pd.read_csv('year_items.csv') items['value'] = items['quantity'] * items['unit_price'] orders['basket'] = orders['id'].map(items.groupby('order_id')['value'].sum()) orders = orders[orders['status'] != 'cancelled'] daily = orders.set_index('ordered_at').resample('D')['basket'].sum().rename('revenue').to_frame() daily['day'] = range(len(daily)) daily['weekday'] = daily.index.dayofweek X = daily[['day', 'weekday']] y = daily['revenue'] late = ... forest = RandomForestRegressor(n_estimators=50, random_state=42).fit(...) base = DummyRegressor().fit(...) mae = ... base_mae = ... print(round(mae, 2), round(base_mae, 2))

Check▶ Run

Test

Check yourself

1. A model for “will the customer come back” gives 99% accuracy, and the baseline is 72.5%. What do you do first?

Ship the model, since it is so accurateAdd more features to reach 100%Look for leakage: a feature that contains the answer

2. You fill the missing values in a column with its mean. From which rows do you compute the mean?

From all the rows, so that it is more accurateFrom the training set only: the filling is a step in a PipelineFrom the test set, because the missing values are there

3. After drop_duplicates the accuracy of the tree falls from 89.4% to 58.4%. Which number is right?

The average of the two58.4%: 89.4% comes from test rows whose twins are in the training set89.4%: drop_duplicates throws away useful data

4. You predict the orders per day for the next three months. How do you split the data?

At random, with stratifyBy time: train on the earlier days, test on the last onesAt random, but with more trees in the forest

5. Why do you put the scaler and the model in one Pipeline, instead of scaling the table once at the start?

Because LogisticRegression does not work without a PipelineSo that the preparation learns only from the data the model learns fromBecause the model is more accurate that way

Check

Start the lesson over

Related lessons

• Builds on: Describing the data · Models: learning from examples · Linear regression · Classification
• Continues in: Cross-validation
• Tasks on this topic: Models with scikit-learn