Skip to content
wannacode

Linear regression

In short: Linear regression predicts a number with a line y = w · x + b, and the mean squared error (MSE) says in one number how well the line fits the data.

Problem

Two colleagues, two lines

The warehouse wants to promise hours, not days. Here are 10 past orders: the distance and how many hours they took.

# past orders: distance in km and hours to delivery
KM = [30, 50, 80, 100, 130, 160, 200, 240, 270, 300]
HOURS = [10, 11, 17, 17, 23, 24, 31, 33, 40, 44]

Two colleagues look at the points and each proposes a rule:

  • line A: each kilometre adds 0.08 hours, plus 12 hours in the warehouse;
  • line B: each kilometre adds 0.12 hours, plus 6 hours in the warehouse.

On the chart both look decent. The colleagues argue all afternoon. The argument will not end until they have one number that says how good a line is.

The idea

The line as a model

Both proposals are of one kind: hours = w · km + b.

  • w is the slope: how many hours each kilometre adds;
  • b is the intercept: the hours at 0 km, that is the handling in the warehouse.

A model that predicts a number with a line is called linear regression. Training the model means choosing w and b.

# the line: hours = w · km + b
def predict(w, b, km):
    return w * km + b

Unlike k nearest neighbours, the line does not keep the orders. It is two numbers, and a forecast is one multiplication and one addition. And it continues its slope: for 500 km it says 0.12 · 500 + 6 = 66 hours, while the neighbours of 500 km are the orders from 300 km.

The idea

The error of each order

For every order the line says some number. The error is the difference: what the line says minus what really happened. It is positive when the line promises more, negative when it promises less.

Line B, order by order:

w, b = 0.12, 6
for km, hours in zip(KM, HOURS):
    guess = predict(w, b, km)
    print(f"{km} km: actual {hours} h, the line {guess:.2f} h, error {guess - hours:.2f}")
print(f"MSE: {mse(w, b):.2f}")
Output
30 km: actual 10 h, the line 9.60 h, error -0.40
50 km: actual 11 h, the line 12.00 h, error 1.00
80 km: actual 17 h, the line 15.60 h, error -1.40
100 km: actual 17 h, the line 18.00 h, error 1.00
130 km: actual 23 h, the line 21.60 h, error -1.40
160 km: actual 24 h, the line 25.20 h, error 1.20
200 km: actual 31 h, the line 30.00 h, error -1.00
240 km: actual 33 h, the line 34.80 h, error 1.80
270 km: actual 40 h, the line 38.40 h, error -1.60
300 km: actual 44 h, the line 42.00 h, error -2.00
MSE: 1.83

Ten errors, some plus, some minus. We need one number for the whole line.

The idea

One number: the mean square

Each error is squared, the squares are added and divided by the number of orders. This is the mean squared error, MSE. A number that says how bad a model is, is called a loss function (loss). Less is better, and 0 means the line goes through every point.

# the mean of the squared errors (MSE)
def mse(w, b):
    total = 0
    for km, hours in zip(KM, HOURS):
        error = predict(w, b, km) - hours
        total += error * error
    return total / len(KM)
zip(KM, HOURS) walks both lists at once: at each step it gives one pair of km and hours.

Why a square? Compare three lines. C is the flat line through the mean, 25 hours for every order:

# the mean error without the square
def mean_error(w, b):
    total = 0
    for km, hours in zip(KM, HOURS):
        total += predict(w, b, km) - hours
    return total / len(KM)
lines = {"A": (0.08, 12), "B": (0.12, 6), "C": (0, 25)}
for name, (w, b) in lines.items():
    print(f"line {name}: MSE {mse(w, b):.2f}, mean error {mean_error(w, b):.2f}")
Output
line A: MSE 17.35, mean error -0.52
line B: MSE 1.83, mean error -0.28
line C: MSE 124.00, mean error 0.00

Without the square line C has a mean error of 0.00: it is above the points as much as below them, and the pluses cancel the minuses. The worst line "wins". With the square nothing cancels: C is 124, A is 17.35, and B is 1.83. The colleague with line B wins the argument.

The square does something else too: a big error weighs more. An error of 4 hours gives 16, while two errors of 2 hours give 8 in total. A line that is a little wrong everywhere is better than one that is exact on half the orders and very wrong on the other half.

See it

The errors as squares

def predict(w, b, km):
    return w * km + b




def mse(w, b):
    total = 0
    for km, hours in zip(KM, HOURS):
        error = predict(w, b, km) - hours
        total += error * error
    return total / len(KM)
Step by step, in words
  1. 10 past orders: how many km, and how many hours until delivery.
  2. Line A: 0.08 hours per km plus 12 hours.
  3. The error of each order: what the line says minus what really happened. Some points are above the line, some below.
  4. Each error squared is a square with the error as its side. The mean area is the MSE: 17.35.
  5. Line B: 0.12 hours per km plus 6 hours. The line turns, and the squares shrink.
  6. The MSE of B is 1.83, against 17.35 for A.
  7. The line with the smaller MSE is closer to the points. One number instead of an argument.

Each square is one error squared: its side is the distance from the point to the line. The MSE is their mean area. Line B is better because its squares are smaller, not because someone thought so.

Try it

Predict the output

The courier charges €2 per kilogram. Three parcels of 1, 2 and 3 kg cost €3, €5 and €7. What will the program print?

kg = [1, 2, 3]
price = [3, 5, 7]
total = 0
for x, y in zip(kg, price):
    error = 2 * x - y
    total += error * error
print(f"{total / 3:.2f}")
Answer

Try it

Fill in

What is added to the sum for each order?

error = predict(w, b, km) - hours
total +=

Try it

Find the bug

A colleague writes the loss function and compares the lines. Their program says the best one is the flat line C. Which line has the bug?

Pick the line with the bug.

The idea

When the line lies

The line is the simplest model and is often enough. But:

  • The world is not straight. If after 300 km the order goes through another warehouse and waits another day, the points make a step. The line goes between the two parts and is wrong on both.
  • One odd point. An order lost for a week is an error of hundreds of hours, and its square weighs as much as all the others together. The line tilts towards it and starts lying about the normal orders. Such points are checked and removed, or you measure with the mean absolute error.
  • Far outside the data. The orders go up to 300 km. For 2,000 km the line says 246 hours, but nobody has checked whether such a delivery even exists.
  • A relation is not a cause. The line says that farther orders travel longer. It does not say why. If the farther towns are simply served by a slower courier, a faster courier there will break the line.

The line the colleagues picked by eye is just one of countless w and b. Which one is best, and how to find it without guessing, is in the lesson on gradient descent.

More precisely, if you are curious

The formula for the best line. For a line there is an exact formula (the method of least squares). w is how much km and hours move together, divided by how much km move on their own. b is such that the line goes through the mean point:

w = Σ (km − mean km) · (hours − mean hours) / Σ (km − mean km)², b = mean hours − w · mean km.

n = len(KM)
mean_km = 0
mean_hours = 0
for km, hours in zip(KM, HOURS):
    mean_km += km
    mean_hours += hours
mean_km /= n
mean_hours /= n
top = 0
bottom = 0
for km, hours in zip(KM, HOURS):
    top += (km - mean_km) * (hours - mean_hours)
    bottom += (km - mean_km) * (km - mean_km)
w = top / bottom
b = mean_hours - w * mean_km
print(f"the best line: {w:.4f} h/km, {b:.2f} h")
print(f"MSE: {mse(w, b):.2f}")
Output
the best line: 0.1241 h/km, 5.64 h
MSE: 1.62

The best line is slightly better than B (1.62 against 1.83). The colleague judged well.

Why not always the formula? It is for a line with the squared error. With many features, with another loss function and with neural networks there is no exact formula, or it is too slow. That is why gradient descent is used.

More features. The line can have one w per feature: hours = w₁ · km + w₂ · kg + w₃ · parcels + b. The idea and the MSE are the same.

The mean absolute error (MAE) adds |error| instead of the square. It does not punish a big error as much and is calmer about odd points. The MSE is smoother and so more convenient for gradient descent.

In the libraries: LinearRegression in scikit-learn computes exactly this formula. In ML.NET it is the Sdca or Ols regression.

Exercise

The courier's price

Solve the task in your editor.

The courier does not publish its price list, but the shop keeps what it paid for eight parcels:

# parcels: weight in kg and the courier price in €
KG = [0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 5.0, 6.0]
EUR = [4.10, 4.60, 5.00, 5.60, 6.40, 7.50, 8.30, 9.40]

The colleagues propose three lines for the price: A is €0.90 per kg plus €3.80, B is €1.10 per kg plus €3.20, C is €0.70 per kg plus €4.60. Write the loss function (MSE), print it for all three and say which one is best.

Output
line A (€0.90/kg + €3.80): MSE 0.0144
line B (€1.10/kg + €3.20): MSE 0.0769
line C (€0.70/kg + €4.60): MSE 0.2869
smallest MSE: line A

More tasks

  • The mean absolute error. Replace the square with the absolute value. Does the winner change?
  • The formula. Compute the best line with the formula from "More precisely". How much better than A is it?

Test

Check yourself

1. The line is hours = 0.12 · km + 6. What does 6 mean?
2. Why are the errors squared before they are added?
3. Line A has an MSE of 17.35 and line B an MSE of 1.83. Which is better for these orders?
4. One order gets lost and arrives after 200 hours instead of 25. What happens to the line with the smallest MSE?
5. Why can the line say something about an order of 500 km, while k nearest neighbours cannot?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·