Skip to content
wannacode

Training from start to finish

In short: Training a neural network is a loop over epochs: shuffle, batches, a forward pass, backpropagation and a step, until the validation error stops falling.

Problem

The network says −0.23 days

The checkout page should say “arrives within 2 days”. The delivery time depends on the distance, on whether the order is after 4 pm (then it leaves the next day) and a little on the weight. A straight line from linear regression does not follow this curve: in the city it is about a day, up to 200 km it grows fast, then slower and slower.

The network from the backprop lesson can follow a curve. Here it is with 4 hidden neurons and random weights, before training:

rng = Rng(42)
net = Net(4, 3)
net.randomize(rng, 1.0)
print(f"error (MSE): {net.loss(train_x, train_y):.4f}")
for k in [0, 12, 23]:
    print(f"{TRAIN[k][0]} km: the network says {net.predict(train_x[k]):.2f}, "
          f"truth {train_y[k]:.1f} days")
Output
error (MSE): 7.1855
5 km: the network says -0.23, truth 1.0 days
150 km: the network says -0.17, truth 2.3 days
520 km: the network says 0.18, truth 3.0 days

An order 5 km away arrives in −0.23 days. In the backprop lesson you saw one step for one example. How many steps do you need, with which examples, in what order, and when do you stop?

The idea

The training loop

Training is one loop that keeps turning until the network stops improving:

  1. Initial weights: small random numbers from a seeded generator.
  2. Epoch: one pass over all the training examples. Before each epoch the examples are shuffled, so the network does not learn their order.
  3. Batch: a few examples at once. Each gets a forward pass and an error, then a backward pass. The slopes are averaged and the network takes one step.
  4. Check: every 100 epochs the program prints the error on training and on validation.

This is the training loop. The numbers it prints, drawn as a chart, are the error curve (loss curve). From it you can tell whether the network is learning, whether the rate is good and when to stop.

The data is 32 orders from the shop: 24 for training and 8 for validation. The network never takes a step with the validation orders. They are the measure for new orders.

See it

The data and the network

The orders, written in the code:

# order: km, after 4 pm (1 or 0), kg, days to delivery
TRAIN = [
    (5, 0, 1.2, 1.0), (12, 1, 3.5, 1.6), (18, 0, 0.8, 1.0),
    (25, 0, 6.0, 1.2), (28, 1, 2.1, 1.8), (35, 0, 4.4, 1.3),
    (48, 1, 0.5, 2.2), (60, 0, 9.0, 2.5), (75, 0, 1.7, 1.6),
    (90, 1, 2.9, 2.6), (110, 0, 5.5, 1.9), (130, 1, 0.9, 2.7),
    (150, 0, 3.1, 2.3), (170, 0, 8.2, 3.3), (185, 1, 1.4, 3.1),
    (195, 0, 2.6, 2.3), (210, 0, 4.8, 2.6), (240, 1, 0.6, 3.3),
    (280, 0, 8.6, 3.6), (320, 0, 1.9, 2.7), (360, 1, 3.3, 3.6),
    (410, 0, 5.0, 2.8), (460, 1, 2.2, 3.6), (520, 0, 0.7, 3.0),
]
# validation: orders the network does not see while training
VALID = [
    (8, 1, 2.4, 1.6), (22, 0, 8.8, 1.0), (40, 0, 1.1, 1.4),
    (95, 1, 7.9, 2.5), (160, 0, 2.0, 2.3), (205, 1, 3.9, 3.1),
    (300, 0, 9.4, 2.6), (480, 1, 4.6, 3.6),
]

The features are brought to similar scales: kilometres in hundreds, weight in tens of kilograms. Otherwise 520 km would drown out everything else:

# features on similar scales: hundreds of km, 0 or 1, tens of kg
def inputs(orders):
    return [[km / 100, late, kg / 10] for km, late, kg, _ in orders]




def targets(orders):
    return [days for *_, days in orders]

The randomness comes from our own seeded generator. One seed gives the same numbers on every computer and in all four languages, so the lesson prints the same output everywhere. The number 48271 and the division by 2³¹ − 1 are an old, well-tested generator (Park–Miller). The product stays below 2⁵³, so it is exact even in JavaScript:

class Rng:
    def __init__(self, seed):
        self.state = seed


    # the next number between 0 and 1; one seed gives the same numbers
    def next(self):
        self.state = self.state * 48271 % 2147483647
        return self.state / 2147483647
random.seed(42) in Python also gives repeatable numbers. Here the generator is ours, so it is the same as in the other three languages.

The network is the one from the backprop lesson, put into a class: 3 inputs, 4 hidden neurons with a sigmoid and one output without an activation, because days are not a probability. step does a forward and a backward pass for every order of the batch, adds up the slopes and moves each weight once, by the average slope:

def sigmoid(z):
    return 1 / (1 + math.exp(-z))




class Net:
    def __init__(self, hidden, inputs):
        self.w1 = [[0.0] * inputs for _ in range(hidden)]  # input → hidden layer
        self.b1 = [0.0] * hidden
        self.w2 = [0.0] * hidden  # hidden layer → output
        self.b2 = 0.0
        self.h = [0.0] * hidden  # the hidden outputs from the last predict


    # random weights from -scale to scale; biases stay 0
    def randomize(self, rng, scale):
        for row in self.w1:
            for i in range(len(row)):
                row[i] = (rng.next() * 2 - 1) * scale
        for j in range(len(self.w2)):
            self.w2[j] = (rng.next() * 2 - 1) * scale


    def predict(self, x):
        y = self.b2
        for j, row in enumerate(self.w1):
            z = self.b1[j]
            for i in range(len(x)):
                z += row[i] * x[i]
            self.h[j] = sigmoid(z)
            y += self.w2[j] * self.h[j]
        return y


    def loss(self, xs, ys):
        total = 0.0
        for x, y in zip(xs, ys):
            e = self.predict(x) - y
            total += e * e
        return total / len(xs)


    # one step: the average slope over the examples in the batch
    def step(self, xs, ys, batch, rate):
        hidden, inputs = len(self.w2), len(xs[0])
        gw1 = [[0.0] * inputs for _ in range(hidden)]
        gb1, gw2, gb2 = [0.0] * hidden, [0.0] * hidden, 0.0
        for k in batch:
            d = 2 * (self.predict(xs[k]) - ys[k])
            gb2 += d
            for j in range(hidden):
                gw2[j] += d * self.h[j]
                dz = d * self.w2[j] * self.h[j] * (1 - self.h[j])
                gb1[j] += dz
                for i in range(inputs):
                    gw1[j][i] += dz * xs[k][i]
        n = len(batch)
        self.b2 -= rate * gb2 / n
        for j in range(hidden):
            self.w2[j] -= rate * gw2[j] / n
            self.b1[j] -= rate * gb1[j] / n
            for i in range(inputs):
                self.w1[j][i] -= rate * gw1[j][i] / n

The shuffle is Fisher–Yates: from the back to the front each place swaps with a random place before it.

# Fisher–Yates: every order is equally likely
def shuffle(order, rng):
    for i in range(len(order) - 1, 0, -1):
        k = int(rng.next() * (i + 1))
        order[i], order[k] = order[k], order[i]

See it

The loop and the error curve

rng = Rng(42)
net = Net(4, 3)
net.randomize(rng, 1.0)
size = 8
rate = 0.05
order = list(range(len(train_x)))
print("epoch  training  validation")
for epoch in range(1, 1001):
    shuffle(order, rng)
    for start in range(0, len(order), size):
        batch = order[start:start + size]
        net.step(train_x, train_y, batch, rate)
    if epoch % 100 == 0:
        t = net.loss(train_x, train_y)
        v = net.loss(valid_x, valid_y)
        print(f"{epoch:5d}{t:10.4f}{v:12.4f}")
Step by step, in words
  1. Seed 42: every run starts from the same random weights. Before training the error is 7.19, far above the chart.
  2. An epoch is one pass over all 24 training orders. First they are shuffled, with the same generator.
  3. A batch of 8 orders: forward, error, backward, one step. Three steps per epoch, 3000 in the whole training.
  4. Epoch 100: training 0.1365, validation 0.0756. The validation orders were never used for a step.
  5. Validation is lowest at epoch 200: 0.0705. After that it turns up, while training keeps falling.
  6. Epoch 1000: training 0.0381, validation 0.2371. The network memorises three heavy parcels that were late by chance.
  7. The best network is near epoch 200, not 1000. You stop when validation stops falling.

The whole loop: 1000 epochs, batches of 8, rate 0.05:

rng = Rng(42)
net = Net(4, 3)
net.randomize(rng, 1.0)
size = 8
rate = 0.05
order = list(range(len(train_x)))
print("epoch  training  validation")
for epoch in range(1, 1001):
    shuffle(order, rng)
    for start in range(0, len(order), size):
        batch = order[start:start + size]
        net.step(train_x, train_y, batch, rate)
    if epoch % 100 == 0:
        t = net.loss(train_x, train_y)
        v = net.loss(valid_x, valid_y)
        print(f"{epoch:5d}{t:10.4f}{v:12.4f}")
Output
epoch  training  validation
  100    0.1365      0.0756
  200    0.1041      0.0705
  300    0.0826      0.0776
  400    0.0667      0.1048
  500    0.0556      0.1350
  600    0.0486      0.1718
  700    0.0443      0.1967
  800    0.0417      0.2202
  900    0.0396      0.2243
 1000    0.0381      0.2371

Training falls from 7.19 to 0.0381. Validation falls until epoch 200 and then climbs: at epoch 1000 it is 0.2371, more than three times worse than the best. The network keeps fitting the training orders, including what in them is chance. We will come back to that at the end of the lesson.

Try it

Predict the output

If the batch is 10 and there are 24 examples, how many steps does training take in 1000 epochs?

examples, size, steps = 24, 10, 0
for epoch in range(1, 1001):
    for start in range(0, examples, size):
        steps += 1
print(f"steps: {steps}")
Answer

The idea

Batches: all, one at a time or 8

How many examples should go into one step? Here is the same network for 40 epochs with a batch of 24 (all at once), 8 and 1. After every step the program measures the error over all 24 orders and counts how many times the step raised it:

for size in [24, 8, 1]:
    rng = Rng(42)
    net = Net(4, 3)
    net.randomize(rng, 1.0)
    order = list(range(len(train_x)))
    steps, up = 0, 0
    before = net.loss(train_x, train_y)
    for epoch in range(1, 41):
        shuffle(order, rng)
        for start in range(0, len(order), size):
            net.step(train_x, train_y, order[start:start + size], 0.05)
            after = net.loss(train_x, train_y)
            steps += 1
            if after > before:
                up += 1
            before = after
    print(f"batch {size:2d}: {steps:3d} steps, {up:3d} up, error {before:.4f}")
Output
batch 24:  40 steps,   0 up, error 0.2913
batch  8: 120 steps,  21 up, error 0.1840
batch  1: 960 steps, 495 up, error 0.1238
  • All at once: the slope is the exact slope of the whole error, so not one step raises it. But there is one step per epoch, and after 40 epochs the network is the furthest away.
  • One example at a time: 24 steps per epoch, so in 40 epochs it gets the lowest. But the slope of one example is a rough guess of the slope of all: 495 of 960 steps raise the error.
  • A batch of 8: three steps per epoch and only 21 of 120 up.

This is called a mini-batch: a step by the average slope of a few examples. The batch is the compromise almost everyone uses. On a graphics card the examples of one batch are computed at once, so in real projects batches are tens or hundreds of examples.

Try it

Fill in the blank

The slopes of the batch are added up, and the step goes by the average slope. What do you divide by?

n = 
self.b2 -= rate * gb2 / n

The idea

A bad rate, a bad start

The curve tells you what is wrong. The three rates from the gradient descent lesson, but in a network. The error is at epochs 50, 100 and 200:

for rate in [0.005, 0.05, 2.0]:
    rng = Rng(42)
    net = Net(4, 3)
    net.randomize(rng, 1.0)
    order = list(range(len(train_x)))
    line = f"rate {rate:.3f}:"
    for epoch in range(1, 201):
        train_epoch(net, train_x, train_y, order, 8, rate, rng)
        loss = net.loss(train_x, train_y)
        if math.isnan(loss) or loss > 1e6:
            line += f" blew up at epoch {epoch}"
            break
        if epoch in (50, 100, 200):
            line += f" {loss:.4f}"
    print(line)
Output
rate 0.005: 0.3892 0.3201 0.2472
rate 0.050: 0.1698 0.1365 0.1041
rate 2.000: blew up at epoch 2

With 0.005 the network learns, but slowly: at epoch 200 (0.2472) it is worse than 0.05 at epoch 50 (0.1698). The curve falls gently and does not stop falling, which means you can raise the rate. With 2 every step overshoots the bottom and the next slope is even steeper. The error is over a million already in the second epoch, and after it come infinity and NaN. That is why the program checks and stops.

The start matters too:

rng = Rng(42)
zeros = Net(4, 3)  # no randomize: every weight is 0
order = list(range(len(train_x)))
for epoch in range(1, 201):
    train_epoch(zeros, train_x, train_y, order, 8, 0.05, rng)
print(f"zeros: error {zeros.loss(train_x, train_y):.4f}")
for j, row in enumerate(zeros.w1):
    print(f"  neuron {j + 1}: " + " ".join(f"{x:.2f}" for x in row))


for scale in [8, 1]:
    rng = Rng(42)
    net = Net(4, 3)
    net.randomize(rng, scale)
    first = net.loss(train_x, train_y)
    order = list(range(len(train_x)))
    for epoch in range(1, 201):
        train_epoch(net, train_x, train_y, order, 8, 0.05, rng)
    print(f"weights ×{scale}: at the start {first:.4f}, after 200 epochs {net.loss(train_x, train_y):.4f}")
Output
zeros: error 0.1127
  neuron 1: 0.99 0.47 0.19
  neuron 2: 0.99 0.47 0.19
  neuron 3: 0.99 0.47 0.19
  neuron 4: 0.99 0.47 0.19
weights ×8: at the start 33.9817, after 200 epochs 0.2971
weights ×1: at the start 7.1855, after 200 epochs 0.1041
  • Zeros. The four hidden neurons start identical, get identical slopes and take identical steps. After 200 epochs their weights are the same to the last digit: the network has one hidden neuron, repeated four times. Random initial weights are there so the neurons become different.
  • Weights ×8. The sums in the sigmoid are large, so for most orders it is almost 0 or almost 1, where it is flat. A flat curve means an almost zero slope and tiny steps: after 200 epochs the error is almost three times bigger than with weights ×1.

Try it

Find the bug

A colleague writes the loop their own way and shuffles the arrays themselves instead of the indices. The error stays above 0.6 and does not fall, however many epochs they run. Which line is the bug on?

Pick the line with the bug.

The idea

When to stop

What exactly does the network memorise? Among the training orders there are three heavy parcels (9.0 kg, 8.2 kg and 8.6 kg) that were a whole day late by chance. Here are the three heavy parcels from validation, which were not late:

rng = Rng(42)
net = Net(4, 3)
net.randomize(rng, 1.0)
order = list(range(len(train_x)))
for epoch in range(1, 1001):
    train_epoch(net, train_x, train_y, order, 8, 0.05, rng)
    if epoch in (200, 1000):
        print(f"epoch {epoch}:")
        for k in [1, 3, 6]:  # the heavy parcels from validation
            km, _, kg, days = VALID[k]
            print(f"  {km} km, {kg:.1f} kg: {net.predict(valid_x[k]):.2f} (truth {days:.1f})")
Output
epoch 200:
  22 km, 8.8 kg: 1.31 (truth 1.0)
  95 km, 7.9 kg: 2.53 (truth 2.5)
  300 km, 9.4 kg: 3.13 (truth 2.6)
epoch 1000:
  22 km, 8.8 kg: 1.84 (truth 1.0)
  95 km, 7.9 kg: 3.09 (truth 2.5)
  300 km, 9.4 kg: 3.38 (truth 2.6)

At epoch 200 the predictions are close to the truth. At epoch 1000 the network has learned “heavy travels slower”: the parcel 22 km away gets 1.84 days instead of 1.0. A rule that is only chance in the data is overfitting: training gets better, and new orders get worse.

That is why you stop when validation stops falling. This is early stopping. The program checks validation every 10 epochs and keeps a snapshot (a copy) of the best network. If there is no improvement for 100 epochs, it stops and returns the snapshot:

# a copy of the weights: a snapshot of the network right now
def copy(self):
    c = Net(len(self.w2), len(self.w1[0]))
    c.w1 = [row[:] for row in self.w1]
    c.b1, c.w2, c.b2 = self.b1[:], self.w2[:], self.b2
    return c
rng = Rng(42)
net = Net(4, 3)
net.randomize(rng, 1.0)
order = list(range(len(train_x)))
# check every 10 epochs; stop after 100 with no improvement
best, best_epoch, epoch = math.inf, 0, 0
best_net = net.copy()
while epoch < 1000 and epoch - best_epoch < 100:
    epoch += 1
    train_epoch(net, train_x, train_y, order, 8, 0.05, rng)
    if epoch % 10 == 0:
        valid = net.loss(valid_x, valid_y)
        if valid < best:
            best, best_epoch = valid, epoch
            best_net = net.copy()
print(f"lowest validation: {best:.4f} at epoch {best_epoch}")
print(f"stopped at epoch {epoch}")
print(f"keeping the network from epoch {best_epoch}: {best_net.loss(valid_x, valid_y):.4f}")
Output
lowest validation: 0.0667 at epoch 230
stopped at epoch 330
keeping the network from epoch 230: 0.0667

The best validation is 0.0667 at epoch 230, better than any point on the curve above, because the check is more frequent. Training stops at epoch 330 instead of 1000, and the network kept is the one from epoch 230.

Validation decided when to stop, so it is no longer an honest measure for new orders. The final score is taken on a third part of the data, the test set, which took part in no decision.

The idea

When it hurts

  • A network for everything. 32 orders and three features are little data: a line with a few features or a simpler formula is often enough, is easier to explain and does not memorise. A network pays off when the rules are too many to write and there are enough examples.
  • One run. Another seed gives other initial weights and another curve. One run may be luck. Compare changes with several seeds and look at the average.
  • A small validation set. Eight orders are a noisy measure: one odd order moves the curve. The less data you have, the more carefully you read a single point.
  • Early stopping does not fix the data. The three chance delays are still there. It is better to find them and remove or fix them, and if the weight really has no effect, not to give it as a feature.
  • The cost. Here training takes under a second. Real networks train for hours or weeks on many graphics cards. Training clusters and putting a model into production are infrastructure and are taught in the DevOps course ZeroToOps.
More precisely, if you are curious

The step as a formula. The batch has B examples, and η is the rate. For every weight w:

w ← w − η · (1/B) · Σ ∂L/∂w

In words: add up the slope of the error with respect to w for every example in the batch, divide by B, multiply by the rate and subtract. ∂L/∂w is the slope backprop computes for one weight. When B is all the examples, this is ordinary gradient descent. With B = 1 it is called stochastic (SGD), and with batches it is mini-batch SGD.

Why 2 · (y − t). The error of one example is (y − t)². Its slope with respect to y is 2 · (y − t): if the output is 0.5 above the truth, the slope is 1, and if it is 0.5 below, it is −1. That is where backprop starts in step.

The real tools. Machine learning practice is mostly in Python. In PyTorch the same loop is: DataLoader(..., batch_size=8, shuffle=True) gives the batches, optimizer.zero_grad() resets the slopes (our new arrays at the start of the step), loss.backward() is backprop, and optimizer.step() moves the weights. The seed is torch.manual_seed(42). In C# the same is done by ML.NET or TorchSharp, in Java by DJL or Tribuo, and in JavaScript by TensorFlow.js, while ONNX Runtime runs an already trained model. The loop is the same, only the matrices are computed on a graphics card.

Smarter steps. Almost nobody uses the bare step from the lesson. Momentum remembers the direction of the last steps and goes through the noise of the batches more smoothly. Adam keeps a separate rate for every weight. The rate is often lowered over time (a learning rate schedule): big steps at the start, small ones at the end.

The initial weights. Here the random weights are from −1 to 1. In bigger networks the range is computed from the number of inputs of the neuron, for example ±1/√n (Xavier, He), so the sums are not too large already in the first layer.

The same numbers in four languages. The additions and multiplications here go in the same order, so the result is the same bit for bit. exp is not promised to be identical to the last bit in every language, so the programs print 4 decimals. The output is checked and is the same in Java, C#, Python and JavaScript.

f"{x:.4f}" always writes a point, whatever the language of the system.

Exercise

Early stopping at three rates

Solve the task in your own editor.

Take early stopping from the lesson and run it for rates 0.02, 0.05 and 0.2, each time with a new network from seed 42. For each rate print the lowest validation, its epoch and the epoch where training stops. You do not need to keep a copy of the network.

Output
rate 0.02: lowest validation 0.0683 at epoch 470, stopped at epoch 570
rate 0.05: lowest validation 0.0667 at epoch 230, stopped at epoch 330
rate 0.20: lowest validation 0.0628 at epoch 50, stopped at epoch 150

With 0.2 the network reaches its lowest validation in 50 epochs and stops at 150. With 0.02 it needs 470. The bigger rate reaches the bottom sooner, but it also turns up sooner. That is why validation is checked often, not only at the end.

More tasks

  • Six neurons. Run the main loop with 6 hidden neurons. Does validation turn up earlier?
  • Without the weight. Remove the weight from the features (2 inputs instead of 3) and run the main loop. What happens to validation at epoch 1000?

Test

Check yourself

1. What is one epoch?
2. The training error falls the whole time, while validation is lowest at epoch 200 and then grows. Which network do you keep?
3. After the second epoch the error is over a million, and then it becomes NaN. What is most likely?
4. All the weights are 0 at the start. What happens to the four hidden neurons?
5. Why do you usually train with batches, not with all the examples at once or one at a time?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·