Skip to content
wannacode

What machine learning is

In short: Machine learning is a way for a program to learn the rule from examples (features and a label) instead of a programmer writing it by hand with ifs.

Problem

How many days will the order take

The product page must say "arrives in 2 days". A colleague writes the rule by kilometres: up to 100 km it is 1 day, up to 250 km it is 2 days, and beyond that 3. Then they run it on 12 past orders whose travel time we know:

// a colleague's rule: by kilometres
static int rule(double km) {
    if (km < 100) return 1;
    if (km < 250) return 2;
    return 3;
}
int right = 0;
for (int i = 0; i < ORDERS.length; i++) {
    int guess = rule(ORDERS[i][0]);
    if (guess == DAYS[i]) right++;
    else System.out.println("order " + (i + 1) + ": the rule says " + guess + ", it took " + DAYS[i]);
}
System.out.println("the rule gets right " + right + " of " + ORDERS.length);
Output
order 2: the rule says 1, it took 2
order 5: the rule says 2, it took 3
order 6: the rule says 2, it took 3
order 9: the rule says 3, it took 4
order 10: the rule says 3, it took 4
order 12: the rule says 3, it took 5
the rule gets right 6 of 12

Six out of twelve. The colleague adds two rules: a heavy parcel (over 10 kg) takes a day longer, and so does a busy courier (over 35 parcels that day). The same loop, with the new rule:

// two more rules: a heavy parcel and a busy courier
static int rule(double km, double kg, double parcels) {
    int days = km < 100 ? 1 : km < 250 ? 2 : 3;
    if (kg > 10) days++;
    if (parcels > 35) days++;
    return days;
}
Output
order 2: the rules say 1, it took 2
order 12: the rules say 4, it took 5
the rules get right 10 of 12

Ten out of twelve, but with three new numbers picked by eye: 10 kg, 35 parcels and "a day longer". Why not 8 kg? Why +1 and not +0.5? And next month the shop changes couriers, and all the numbers are wrong again. With 50,000 orders and ten things that matter, nobody can tune them by hand.

The idea

The rule from the examples

Turn it around. Instead of writing the rule and checking it on the past orders, give the program the past orders and let it find the rule itself. This is called machine learning: a program that derives the rule from examples instead of someone writing it.

Every example has two parts:

  • features: what you know about the order before it leaves. Here there are three: the distance, the weight and how many parcels the courier has that day;
  • label: the answer you want to guess. Here it is the number of days the order took.
// past orders: km, kg, parcels for the courier that day
static final double[][] ORDERS = {
        {15, 1.2, 18}, {40, 0.8, 35}, {60, 3.5, 20},
        {120, 2.0, 22}, {150, 12.0, 25}, {180, 1.5, 40},
        {220, 0.5, 15}, {250, 8.0, 30}, {300, 2.5, 45},
        {320, 18.0, 20}, {380, 1.0, 28}, {400, 6.0, 42},
};
// how many days each one took
static final int[] DAYS = {1, 2, 1, 2, 3, 3, 2, 3, 4, 4, 3, 5};

What you get from the examples, and what gives an answer for a new order, is called a model. There are many kinds of models. We start with the simplest one, where you can see every number.

The idea

The nearest neighbours

How would you guess? You would look for past orders that resemble the new one: a similar distance, a similar weight, a similar load. If they took 3 days, the new one will probably take as long.

That is a model: k nearest neighbours (k-NN). It keeps all past orders. For a new order it finds the k nearest and returns the average of their days. Here k = 3.

"Near" means a small distance between two orders. The difference in each feature is squared, the squares are added, and you take the square root of the sum, like the distance between two points on a map.

There is a trap. Kilometres are in the hundreds, kilograms in single digits. If you add the differences as they are, a difference of 10 kg weighs as much as 10 km, which is nothing, and kilometres decide on their own. So every difference is divided by the range of its feature (the largest minus the smallest value in the data). Then every feature is on one scale: 1 means "from one end to the other".

// how far apart the values of each feature are: max − min
static double[] ranges(double[][] rows) {
    double[] range = new double[rows[0].length];
    for (int j = 0; j < range.length; j++) {
        double min = rows[0][j], max = rows[0][j];
        for (double[] row : rows) {
            min = Math.min(min, row[j]);
            max = Math.max(max, row[j]);
        }
        range[j] = max - min;
    }
    return range;
}
static double distance(double[] a, double[] b) {
    double sum = 0;
    for (int j = 0; j < a.length; j++) {
        double d = (a[j] - b[j]) / RANGE[j];
        sum += d * d;
    }
    return Math.sqrt(sum);
}

See it

The three nearest to the new order

static Integer[] nearest(double[] order) {
    Integer[] near = new Integer[ORDERS.length];
    double[] dist = new double[ORDERS.length];
    for (int i = 0; i < ORDERS.length; i++) {
        near[i] = i;
        dist[i] = distance(order, ORDERS[i]);
    }
    Arrays.sort(near,
            Comparator.comparingDouble(i -> dist[i]));
    return near;
}


static double predict(double[] order, int k) {
    Integer[] near = nearest(order);
    double sum = 0;
    for (int n = 0; n < k; n++) {
        sum += DAYS[near[n]];
    }
    return sum / k;
}
Step by step, in words
  1. A new order: 140 km, 10 kg, 30 parcels for the courier. Below are the 12 past orders and how many days each took.
  2. The distance to every past order. Each difference is divided by the range of its feature: km do not drown out kg.
  3. Then it sorts the orders by distance: the most similar ones come first.
  4. It takes the first k = 3: order 5, order 8, order 4. The rest do not count.
  5. The forecast is the average of their days: (3 + 3 + 2) / 3 = 2.7 days.
  6. Nobody wrote a rule. The answer comes from the past orders that are most like the new one.

The model measures the distance to every past order, sorts them and takes the first k:

static Integer[] nearest(double[] order) {
    Integer[] near = new Integer[ORDERS.length];
    double[] dist = new double[ORDERS.length];
    for (int i = 0; i < ORDERS.length; i++) {
        near[i] = i;
        dist[i] = distance(order, ORDERS[i]);
    }
    Arrays.sort(near,
            Comparator.comparingDouble(i -> dist[i]));
    return near;
}


static double predict(double[] order, int k) {
    Integer[] near = nearest(order);
    double sum = 0;
    for (int n = 0; n < k; n++) {
        sum += DAYS[near[n]];
    }
    return sum / k;
}

The new order is 140 km, 10 kg and 30 parcels for the courier:

double[] order = {140, 10.0, 30};
System.out.printf("the 3 nearest to %.0f km, %.1f kg, %.0f parcels:%n", order[0], order[1], order[2]);
Integer[] near = nearest(order);
for (int n = 0; n < 3; n++) {
    double[] o = ORDERS[near[n]];
    System.out.printf("  order %d: %.0f km, %.1f kg, %.0f parcels → %d days (distance %.2f)%n",
            near[n] + 1, o[0], o[1], o[2], DAYS[near[n]], distance(order, o));
}
System.out.printf("forecast: %.1f days%n", predict(order, 3));
Output
the 3 nearest to 140 km, 10.0 kg, 30 parcels:
  order 5: 150 km, 12.0 kg, 25 parcels → 3 days (distance 0.20)
  order 8: 250 km, 8.0 kg, 30 parcels → 3 days (distance 0.31)
  order 4: 120 km, 2.0 kg, 22 parcels → 2 days (distance 0.53)
forecast: 2.7 days

The nearest is the heavy parcel from 150 km. Order 8 is 100 km further, but it is heavy and on an equally busy day, so it comes second. The light order from 120 km is third. The average of 3, 3 and 2 days is 2.7.

Try it

Predict the output

For an order of 230 km, 3 kg and 24 parcels the three nearest are order 4 (2 days), order 7 (2 days) and order 8 (3 days), in that order. What will the program print?

double[] order = {230, 3.0, 24};
System.out.printf("%.1f %.1f%n", predict(order, 1), predict(order, 3));
Answer

The idea

A forecast for new orders

The model is ready the moment it has the data. New orders:

double[][] fresh = {{140, 10.0, 30}, {90, 1.0, 20}, {350, 4.0, 40}};
for (double[] order : fresh) {
    System.out.printf("%.0f km, %.1f kg, %.0f parcels: %.1f days%n",
            order[0], order[1], order[2], predict(order, 3));
}
Output
140 km, 10.0 kg, 30 parcels: 2.7 days
90 km, 1.0 kg, 20 parcels: 1.3 days
350 km, 4.0 kg, 40 parcels: 4.0 days

How well it guesses, these three numbers do not tell. The honest check is on orders the model has not seen, and it is in the lesson on evaluation.

How do you choose k? With k = 1 the forecast is the days of one single order, and it may be an exception: a holiday, a lost parcel. With k equal to all orders the forecast is the average of everything, and the model stops telling 15 km from 400 km. Usually you try a few values and take the one that is wrong the least on unseen orders.

Try it

Fill in

The forecast is the average of the days of the k nearest. What is the sum divided by?

double sum = 0;
for (int n = 0; n < k; n++) {
    sum += DAYS[near[n]];
}
return sum / ;

Try it

Find the bug

A colleague rewrites the distance. The program runs, but the neighbours of the order of 140 km and 10 kg are now orders 5, 4, 6: the heavy order 8 from the same busy day dropped out, and a light parcel took its place. For the order of 90 km the forecast becomes 1.7 days instead of 1.3. Which line has the bug?

Pick the line with the bug.

The idea

When machine learning is the wrong tool

A model is a rule that nobody wrote. That is both its strength and its cost.

  • The rule is known and stable. The delivery fee is €4.90 up to 2 kg, €6.90 above 2 kg, free above €50. That is an if, not a model. A model would learn it only roughly and would sometimes be off by a few cents.
  • You need an exact answer or an explanation. Tax, a discount from a contract, the right to return. "Usually it is like this" is not enough there, and "the model decided" is not an explanation for the customer.
  • There is no data. A new shop with no past orders has nothing to learn from. Then a rule by eye is an honest start, and the data comes later.
  • The world has changed. The model knows only the past. A new courier, holidays, a strike: forecasts from the old orders lie until the model is trained again.
  • A mistake is expensive. A model that is wrong in 1 case of 20 is good for "arrives in 2 days" and bad for "let the payment through". There a rule or a person stands next to it.

Machine learning is for the cases where the rule has too many parts for a person to write it, and there are many examples.

More precisely, if you are curious

What one forecast costs. k-NN learns nothing in advance: for every new order it compares with all past ones and sorts them. With n orders that is at least n distances per forecast. For 12 orders it is nothing, for 10 million it is slow. That is why real libraries use structures such as a k-d tree, and with many features they search approximately. What "grows with n" means is in the lesson on Big-O.

The same search, but for text. Searching by meaning in databases is k nearest neighbours over vectors: the lesson on vectors in the SQL course.

Dividing by the range is called min-max normalisation. Another way is to subtract the mean and divide by the standard deviation. Why the scale of the features matters for other models too, you will see in the lesson on gradient descent.

The real tools. Here everything is loops and arrays, so that you see every number. In practice machine learning is done mostly in Python: NumPy for arrays, scikit-learn for the classic models (there k-NN is KNeighborsRegressor), PyTorch for neural networks. C# has ML.NET, Java has DJL and Tribuo, JavaScript has TensorFlow.js and ONNX Runtime. The ideas are the same.

Training models and running them in production (training servers, deployment, watching whether a model gets stale) is infrastructure and is taught in the DevOps course ZeroToOps.

Exercise

How many minutes to pack

Solve the task in your editor.

The warehouse wants to know how many minutes packing an order will take, so it can plan the staff. The features are the number of items and the total weight, and the label is the minutes. Here are ten past orders:

// past orders: number of items, kg
static final double[][] ORDERS = {
        {1, 0.5}, {2, 1.0}, {3, 4.0}, {5, 2.0}, {6, 8.0},
        {8, 3.0}, {10, 12.0}, {12, 5.0}, {15, 9.0}, {20, 6.0},
};
// minutes to pack
static final int[] MINUTES = {4, 6, 9, 10, 15, 14, 22, 18, 25, 28};

Write k nearest neighbours with k = 3 and division by the range, and forecast two new orders: 4 items and 1.5 kg, and 11 items and 10 kg.

Output
4 items, 1.5 kg: 6.7 min
11 items, 10.0 kg: 20.7 min

More tasks

  • Without the range. Remove the division by the range and see which neighbours come out. Which feature decides now?
  • Weights. Give the neighbours a weight: the nearer one counts more (for example 1 / distance). Does the forecast change?

Test

Check yourself

1. The shop trains a model that says how many days an order will take. What is the label?
2. Why does the distance divide every difference by the range of its feature?
3. What does k nearest neighbours do when a new order comes in?
4. The delivery fee is €4.90 up to 2 kg and €6.90 above 2 kg, and free above €50. How do you compute it?
5. Why is k = 1 risky?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·