Training a neural network in C#: epochs and batches

Problem

The network says −0.23 days

The checkout page should say “arrives within 2 days”. The delivery time depends on the distance, on whether the order is after 4 pm (then it leaves the next day) and a little on the weight. A straight line from linear regression does not follow this curve: in the city it is about a day, up to 200 km it grows fast, then slower and slower.

The network from the backprop lesson can follow a curve. Here it is with 4 hidden neurons and random weights, before training:

var rng = new Rng(42);
var net = new Net(4, 3);
net.Randomize(rng, 1.0);
Console.WriteLine($"error (MSE): {net.Loss(trainX, trainY):F4}");
foreach (int k in new[] { 0, 12, 23 })
    Console.WriteLine($"{train[k][0]} km: the network says {net.Predict(trainX[k]):F2}, " +
        $"truth {trainY[k]:F1} days");

error (MSE): 7.1855
5 km: the network says -0.23, truth 1.0 days
150 km: the network says -0.17, truth 2.3 days
520 km: the network says 0.18, truth 3.0 days

An order 5 km away arrives in −0.23 days. In the backprop lesson you saw one step for one example. How many steps do you need, with which examples, in what order, and when do you stop?

The idea

The training loop

Training is one loop that keeps turning until the network stops improving:

Initial weights: small random numbers from a seeded generator.
Epoch: one pass over all the training examples. Before each epoch the examples are shuffled, so the network does not learn their order.
Batch: a few examples at once. Each gets a forward pass and an error, then a backward pass. The slopes are averaged and the network takes one step.
Check: every 100 epochs the program prints the error on training and on validation.

This is the training loop. The numbers it prints, drawn as a chart, are the error curve (loss curve). From it you can tell whether the network is learning, whether the rate is good and when to stop.

The data is 32 orders from the shop: 24 for training and 8 for validation. The network never takes a step with the validation orders. They are the measure for new orders.

See it

The data and the network

The orders, written in the code:

// order: km, after 4 pm (1 or 0), kg, days to delivery
double[][] train =
[
    [5, 0, 1.2, 1.0], [12, 1, 3.5, 1.6], [18, 0, 0.8, 1.0],
    [25, 0, 6.0, 1.2], [28, 1, 2.1, 1.8], [35, 0, 4.4, 1.3],
    [48, 1, 0.5, 2.2], [60, 0, 9.0, 2.5], [75, 0, 1.7, 1.6],
    [90, 1, 2.9, 2.6], [110, 0, 5.5, 1.9], [130, 1, 0.9, 2.7],
    [150, 0, 3.1, 2.3], [170, 0, 8.2, 3.3], [185, 1, 1.4, 3.1],
    [195, 0, 2.6, 2.3], [210, 0, 4.8, 2.6], [240, 1, 0.6, 3.3],
    [280, 0, 8.6, 3.6], [320, 0, 1.9, 2.7], [360, 1, 3.3, 3.6],
    [410, 0, 5.0, 2.8], [460, 1, 2.2, 3.6], [520, 0, 0.7, 3.0],
];
// validation: orders the network does not see while training
double[][] valid =
[
    [8, 1, 2.4, 1.6], [22, 0, 8.8, 1.0], [40, 0, 1.1, 1.4],
    [95, 1, 7.9, 2.5], [160, 0, 2.0, 2.3], [205, 1, 3.9, 3.1],
    [300, 0, 9.4, 2.6], [480, 1, 4.6, 3.6],
];

The features are brought to similar scales: kilometres in hundreds, weight in tens of kilograms. Otherwise 520 km would drown out everything else:

// features on similar scales: hundreds of km, 0 or 1, tens of kg
static double[][] Inputs(double[][] orders) =>
    orders.Select(o => new[] { o[0] / 100, o[1], o[2] / 10 }).ToArray();

static double[] Targets(double[][] orders) =>
    orders.Select(o => o[3]).ToArray();

The randomness comes from our own seeded generator. One seed gives the same numbers on every computer and in all four languages, so the lesson prints the same output everywhere. The number 48271 and the division by 2³¹ − 1 are an old, well-tested generator (Park–Miller). The product stays below 2⁵³, so it is exact even in JavaScript:

class Rng(long seed)
{
    long state = seed;

    // the next number between 0 and 1; one seed gives the same numbers
    public double Next()
    {
        state = state * 48271 % 2147483647;
        return (double)state / 2147483647;
    }
}

new Random(42) in C# has a seed too. Here the generator is ours, so it is the same as in the other three languages.

The network is the one from the backprop lesson, put into a class: 3 inputs, 4 hidden neurons with a sigmoid and one output without an activation, because days are not a probability. Step does a forward and a backward pass for every order of the batch, adds up the slopes and moves each weight once, by the average slope:

class Net
{
    public readonly double[][] W1;  // input → hidden layer
    public readonly double[] B1;
    public readonly double[] W2;    // hidden layer → output
    public double B2;
    readonly double[] h;            // the hidden outputs from the last predict

    public Net(int hidden, int inputs)
    {
        W1 = new double[hidden][];
        for (int j = 0; j < hidden; j++)
            W1[j] = new double[inputs];
        B1 = new double[hidden];
        W2 = new double[hidden];
        h = new double[hidden];
    }

    // random weights from -scale to scale; biases stay 0
    public void Randomize(Rng rng, double scale)
    {
        foreach (var row in W1)
            for (int i = 0; i < row.Length; i++)
                row[i] = (rng.Next() * 2 - 1) * scale;
        for (int j = 0; j < W2.Length; j++)
            W2[j] = (rng.Next() * 2 - 1) * scale;
    }

    static double Sigmoid(double z) => 1 / (1 + Math.Exp(-z));

    public double Predict(double[] x)
    {
        double y = B2;
        for (int j = 0; j < W2.Length; j++)
        {
            double z = B1[j];
            for (int i = 0; i < x.Length; i++)
                z += W1[j][i] * x[i];
            h[j] = Sigmoid(z);
            y += W2[j] * h[j];
        }
        return y;
    }

    public double Loss(double[][] xs, double[] ys)
    {
        double total = 0;
        for (int k = 0; k < xs.Length; k++)
        {
            double e = Predict(xs[k]) - ys[k];
            total += e * e;
        }
        return total / xs.Length;
    }

    // one step: the average slope over the examples in the batch
    public void Step(double[][] xs, double[] ys, int[] batch, double rate)
    {
        int hidden = W2.Length, inputs = xs[0].Length;
        var gw1 = new double[hidden, inputs];
        double[] gb1 = new double[hidden], gw2 = new double[hidden];
        double gb2 = 0;
        foreach (int k in batch)
        {
            double d = 2 * (Predict(xs[k]) - ys[k]);
            gb2 += d;
            for (int j = 0; j < hidden; j++)
            {
                gw2[j] += d * h[j];
                double dz = d * W2[j] * h[j] * (1 - h[j]);
                gb1[j] += dz;
                for (int i = 0; i < inputs; i++)
                    gw1[j, i] += dz * xs[k][i];
            }
        }
        int n = batch.Length;
        B2 -= rate * gb2 / n;
        for (int j = 0; j < hidden; j++)
        {
            W2[j] -= rate * gw2[j] / n;
            B1[j] -= rate * gb1[j] / n;
            for (int i = 0; i < inputs; i++)
                W1[j][i] -= rate * gw1[j, i] / n;
        }
    }
}

The shuffle is Fisher–Yates: from the back to the front each place swaps with a random place before it.

// Fisher–Yates: every order is equally likely
static void Shuffle(int[] order, Rng rng)
{
    for (int i = order.Length - 1; i > 0; i--)
    {
        int k = (int)(rng.Next() * (i + 1));
        (order[i], order[k]) = (order[k], order[i]);
    }
}

See it

The loop and the error curve

var rng = new Rng(42);
var net = new Net(4, 3);
net.Randomize(rng, 1.0);
int size = 8;
double rate = 0.05;
int[] order = Enumerable.Range(0, trainX.Length).ToArray();
Console.WriteLine("epoch  training  validation");
for (int epoch = 1; epoch <= 1000; epoch++)
{
    Shuffle(order, rng);
    for (int start = 0; start < order.Length; start += size)
    {
        int end = Math.Min(start + size, order.Length);
        int[] batch = order[start..end];
        net.Step(trainX, trainY, batch, rate);
    }
    if (epoch % 100 == 0)
    {
        double t = net.Loss(trainX, trainY);
        double v = net.Loss(validX, validY);
        Console.WriteLine($"{epoch,5}{t,10:F4}{v,12:F4}");
    }
}

Step by step, in words

Seed 42: every run starts from the same random weights. Before training the error is 7.19, far above the chart.
An epoch is one pass over all 24 training orders. First they are shuffled, with the same generator.
A batch of 8 orders: forward, error, backward, one step. Three steps per epoch, 3000 in the whole training.
Epoch 100: training 0.1365, validation 0.0756. The validation orders were never used for a step.
Validation is lowest at epoch 200: 0.0705. After that it turns up, while training keeps falling.
Epoch 1000: training 0.0381, validation 0.2371. The network memorises three heavy parcels that were late by chance.
The best network is near epoch 200, not 1000. You stop when validation stops falling.

The whole loop: 1000 epochs, batches of 8, rate 0.05:

var rng = new Rng(42);
var net = new Net(4, 3);
net.Randomize(rng, 1.0);
int size = 8;
double rate = 0.05;
int[] order = Enumerable.Range(0, trainX.Length).ToArray();
Console.WriteLine("epoch  training  validation");
for (int epoch = 1; epoch <= 1000; epoch++)
{
    Shuffle(order, rng);
    for (int start = 0; start < order.Length; start += size)
    {
        int end = Math.Min(start + size, order.Length);
        int[] batch = order[start..end];
        net.Step(trainX, trainY, batch, rate);
    }
    if (epoch % 100 == 0)
    {
        double t = net.Loss(trainX, trainY);
        double v = net.Loss(validX, validY);
        Console.WriteLine($"{epoch,5}{t,10:F4}{v,12:F4}");
    }
}

epoch  training  validation
  100    0.1365      0.0756
  200    0.1041      0.0705
  300    0.0826      0.0776
  400    0.0667      0.1048
  500    0.0556      0.1350
  600    0.0486      0.1718
  700    0.0443      0.1967
  800    0.0417      0.2202
  900    0.0396      0.2243
 1000    0.0381      0.2371

Training falls from 7.19 to 0.0381. Validation falls until epoch 200 and then climbs: at epoch 1000 it is 0.2371, more than three times worse than the best. The network keeps fitting the training orders, including what in them is chance. We will come back to that at the end of the lesson.

Try it

Predict the output

If the batch is 10 and there are 24 examples, how many steps does training take in 1000 epochs?

int examples = 24, size = 10, steps = 0;
for (int epoch = 1; epoch <= 1000; epoch++)
    for (int start = 0; start < examples; start += size)
        steps++;
Console.WriteLine($"steps: {steps}");

Answer

steps: 1000

steps: 2400

steps: 3000

The idea

Batches: all, one at a time or 8

How many examples should go into one step? Here is the same network for 40 epochs with a batch of 24 (all at once), 8 and 1. After every step the program measures the error over all 24 orders and counts how many times the step raised it:

foreach (int size in new[] { 24, 8, 1 })
{
    var rng = new Rng(42);
    var net = new Net(4, 3);
    net.Randomize(rng, 1.0);
    int[] order = Enumerable.Range(0, trainX.Length).ToArray();
    int steps = 0, up = 0;
    double before = net.Loss(trainX, trainY);
    for (int epoch = 1; epoch <= 40; epoch++)
    {
        Shuffle(order, rng);
        for (int start = 0; start < order.Length; start += size)
        {
            int end = Math.Min(start + size, order.Length);
            net.Step(trainX, trainY, order[start..end], 0.05);
            double after = net.Loss(trainX, trainY);
            steps++;
            if (after > before) up++;
            before = after;
        }
    }
    Console.WriteLine($"batch {size,2}: {steps,3} steps, {up,3} up, error {before:F4}");
}

batch 24:  40 steps,   0 up, error 0.2913
batch  8: 120 steps,  21 up, error 0.1840
batch  1: 960 steps, 495 up, error 0.1238

All at once: the slope is the exact slope of the whole error, so not one step raises it. But there is one step per epoch, and after 40 epochs the network is the furthest away.
One example at a time: 24 steps per epoch, so in 40 epochs it gets the lowest. But the slope of one example is a rough guess of the slope of all: 495 of 960 steps raise the error.
A batch of 8: three steps per epoch and only 21 of 120 up.

This is called a mini-batch: a step by the average slope of a few examples. The batch is the compromise almost everyone uses. On a graphics card the examples of one batch are computed at once, so in real projects batches are tens or hundreds of examples.

Try it

Fill in the blank

The slopes of the batch are added up, and the step goes by the average slope. What do you divide by?

int n = ;
B2 -= rate * gb2 / n;

The idea

A bad rate, a bad start

The curve tells you what is wrong. The three rates from the gradient descent lesson, but in a network. The error is at epochs 50, 100 and 200:

foreach (double rate in new[] { 0.005, 0.05, 2.0 })
{
    var rng = new Rng(42);
    var net = new Net(4, 3);
    net.Randomize(rng, 1.0);
    int[] order = Enumerable.Range(0, trainX.Length).ToArray();
    Console.Write($"rate {rate:F3}:");
    for (int epoch = 1; epoch <= 200; epoch++)
    {
        TrainEpoch(net, trainX, trainY, order, 8, rate, rng);
        double loss = net.Loss(trainX, trainY);
        if (double.IsNaN(loss) || loss > 1e6)
        {
            Console.Write($" blew up at epoch {epoch}");
            break;
        }
        if (epoch is 50 or 100 or 200)
            Console.Write($" {loss:F4}");
    }
    Console.WriteLine();
}

rate 0.005: 0.3892 0.3201 0.2472
rate 0.050: 0.1698 0.1365 0.1041
rate 2.000: blew up at epoch 2

With 0.005 the network learns, but slowly: at epoch 200 (0.2472) it is worse than 0.05 at epoch 50 (0.1698). The curve falls gently and does not stop falling, which means you can raise the rate. With 2 every step overshoots the bottom and the next slope is even steeper. The error is over a million already in the second epoch, and after it come infinity and NaN. That is why the program checks and stops.

The start matters too:

var rng = new Rng(42);
var zeros = new Net(4, 3);   // no Randomize: every weight is 0
int[] order = Enumerable.Range(0, trainX.Length).ToArray();
for (int epoch = 1; epoch <= 200; epoch++)
    TrainEpoch(zeros, trainX, trainY, order, 8, 0.05, rng);
Console.WriteLine($"zeros: error {zeros.Loss(trainX, trainY):F4}");
for (int j = 0; j < zeros.W1.Length; j++)
    Console.WriteLine($"  neuron {j + 1}: " + string.Join(" ", zeros.W1[j].Select(x => x.ToString("F2"))));

foreach (double scale in new double[] { 8, 1 })
{
    rng = new Rng(42);
    var net = new Net(4, 3);
    net.Randomize(rng, scale);
    double first = net.Loss(trainX, trainY);
    order = Enumerable.Range(0, trainX.Length).ToArray();
    for (int epoch = 1; epoch <= 200; epoch++)
        TrainEpoch(net, trainX, trainY, order, 8, 0.05, rng);
    Console.WriteLine($"weights ×{scale}: at the start {first:F4}, after 200 epochs {net.Loss(trainX, trainY):F4}");
}

zeros: error 0.1127
  neuron 1: 0.99 0.47 0.19
  neuron 2: 0.99 0.47 0.19
  neuron 3: 0.99 0.47 0.19
  neuron 4: 0.99 0.47 0.19
weights ×8: at the start 33.9817, after 200 epochs 0.2971
weights ×1: at the start 7.1855, after 200 epochs 0.1041

Zeros. The four hidden neurons start identical, get identical slopes and take identical steps. After 200 epochs their weights are the same to the last digit: the network has one hidden neuron, repeated four times. Random initial weights are there so the neurons become different.
Weights ×8. The sums in the sigmoid are large, so for most orders it is almost 0 or almost 1, where it is flat. A flat curve means an almost zero slope and tiny steps: after 200 epochs the error is almost three times bigger than with weights ×1.

Try it

Find the bug

A colleague writes the loop their own way and shuffles the arrays themselves instead of the indices. The error stays above 0.6 and does not fall, however many epochs they run. Which line is the bug on?

Pick the line with the bug.

The idea

When to stop

What exactly does the network memorise? Among the training orders there are three heavy parcels (9.0 kg, 8.2 kg and 8.6 kg) that were a whole day late by chance. Here are the three heavy parcels from validation, which were not late:

var rng = new Rng(42);
var net = new Net(4, 3);
net.Randomize(rng, 1.0);
int[] order = Enumerable.Range(0, trainX.Length).ToArray();
for (int epoch = 1; epoch <= 1000; epoch++)
{
    TrainEpoch(net, trainX, trainY, order, 8, 0.05, rng);
    if (epoch is 200 or 1000)
    {
        Console.WriteLine($"epoch {epoch}:");
        foreach (int k in new[] { 1, 3, 6 })   // the heavy parcels from validation
            Console.WriteLine($"  {valid[k][0]} km, {valid[k][2]:F1} kg: " +
                $"{net.Predict(validX[k]):F2} (truth {validY[k]:F1})");
    }
}

epoch 200:
  22 km, 8.8 kg: 1.31 (truth 1.0)
  95 km, 7.9 kg: 2.53 (truth 2.5)
  300 km, 9.4 kg: 3.13 (truth 2.6)
epoch 1000:
  22 km, 8.8 kg: 1.84 (truth 1.0)
  95 km, 7.9 kg: 3.09 (truth 2.5)
  300 km, 9.4 kg: 3.38 (truth 2.6)

At epoch 200 the predictions are close to the truth. At epoch 1000 the network has learned “heavy travels slower”: the parcel 22 km away gets 1.84 days instead of 1.0. A rule that is only chance in the data is overfitting: training gets better, and new orders get worse.

That is why you stop when validation stops falling. This is early stopping. The program checks validation every 10 epochs and keeps a snapshot (a copy) of the best network. If there is no improvement for 100 epochs, it stops and returns the snapshot:

// a copy of the weights: a snapshot of the network right now
public Net Copy()
{
    var c = new Net(W2.Length, W1[0].Length);
    for (int j = 0; j < W2.Length; j++)
    {
        W1[j].CopyTo(c.W1[j], 0);
        c.B1[j] = B1[j];
        c.W2[j] = W2[j];
    }
    c.B2 = B2;
    return c;
}

var rng = new Rng(42);
var net = new Net(4, 3);
net.Randomize(rng, 1.0);
int[] order = Enumerable.Range(0, trainX.Length).ToArray();
// check every 10 epochs; stop after 100 with no improvement
double best = double.MaxValue;
int bestEpoch = 0, epoch = 0;
var bestNet = net.Copy();
while (epoch < 1000 && epoch - bestEpoch < 100)
{
    epoch++;
    TrainEpoch(net, trainX, trainY, order, 8, 0.05, rng);
    if (epoch % 10 == 0)
    {
        double loss = net.Loss(validX, validY);
        if (loss < best)
        {
            best = loss;
            bestEpoch = epoch;
            bestNet = net.Copy();
        }
    }
}
Console.WriteLine($"lowest validation: {best:F4} at epoch {bestEpoch}");
Console.WriteLine($"stopped at epoch {epoch}");
Console.WriteLine($"keeping the network from epoch {bestEpoch}: {bestNet.Loss(validX, validY):F4}");

lowest validation: 0.0667 at epoch 230
stopped at epoch 330
keeping the network from epoch 230: 0.0667

The best validation is 0.0667 at epoch 230, better than any point on the curve above, because the check is more frequent. Training stops at epoch 330 instead of 1000, and the network kept is the one from epoch 230.

Validation decided when to stop, so it is no longer an honest measure for new orders. The final score is taken on a third part of the data, the test set, which took part in no decision.

The idea

When it hurts

A network for everything. 32 orders and three features are little data: a line with a few features or a simpler formula is often enough, is easier to explain and does not memorise. A network pays off when the rules are too many to write and there are enough examples.
One run. Another seed gives other initial weights and another curve. One run may be luck. Compare changes with several seeds and look at the average.
A small validation set. Eight orders are a noisy measure: one odd order moves the curve. The less data you have, the more carefully you read a single point.
Early stopping does not fix the data. The three chance delays are still there. It is better to find them and remove or fix them, and if the weight really has no effect, not to give it as a feature.
The cost. Here training takes under a second. Real networks train for hours or weeks on many graphics cards. Training clusters and putting a model into production are infrastructure and are taught in the DevOps course ZeroToOps.

More precisely, if you are curious

The step as a formula. The batch has B examples, and η is the rate. For every weight w:

w ← w − η · (1/B) · Σ ∂L/∂w

In words: add up the slope of the error with respect to w for every example in the batch, divide by B, multiply by the rate and subtract. ∂L/∂w is the slope backprop computes for one weight. When B is all the examples, this is ordinary gradient descent. With B = 1 it is called stochastic (SGD), and with batches it is mini-batch SGD.

Why 2 · (y − t). The error of one example is (y − t)². Its slope with respect to y is 2 · (y − t): if the output is 0.5 above the truth, the slope is 1, and if it is 0.5 below, it is −1. That is where backprop starts in Step.

The real tools. Machine learning practice is mostly in Python. In PyTorch the same loop is: DataLoader(..., batch_size=8, shuffle=True) gives the batches, optimizer.zero_grad() resets the slopes (our new arrays at the start of the step), loss.backward() is backprop, and optimizer.step() moves the weights. The seed is torch.manual_seed(42). In C# the same is done by ML.NET or TorchSharp, in Java by DJL or Tribuo, and in JavaScript by TensorFlow.js, while ONNX Runtime runs an already trained model. The loop is the same, only the matrices are computed on a graphics card.

Smarter steps. Almost nobody uses the bare step from the lesson. Momentum remembers the direction of the last steps and goes through the noise of the batches more smoothly. Adam keeps a separate rate for every weight. The rate is often lowered over time (a learning rate schedule): big steps at the start, small ones at the end.

The initial weights. Here the random weights are from −1 to 1. In bigger networks the range is computed from the number of inputs of the neuron, for example ±1/√n (Xavier, He), so the sums are not too large already in the first layer.

The same numbers in four languages. The additions and multiplications here go in the same order, so the result is the same bit for bit. exp is not promised to be identical to the last bit in every language, so the programs print 4 decimals. The output is checked and is the same in Java, C#, Python and JavaScript.

:F4 writes the number by the culture of the system. On a computer with a Bulgarian culture you will see 0,1365, while the lesson’s check runs with the invariant culture and a point.

Exercise

Early stopping at three rates

Solve the task in your own editor.

Take early stopping from the lesson and run it for rates 0.02, 0.05 and 0.2, each time with a new network from seed 42. For each rate print the lowest validation, its epoch and the epoch where training stops. You do not need to keep a copy of the network.

rate 0.02: lowest validation 0.0683 at epoch 470, stopped at epoch 570
rate 0.05: lowest validation 0.0667 at epoch 230, stopped at epoch 330
rate 0.20: lowest validation 0.0628 at epoch 50, stopped at epoch 150

With 0.2 the network reaches its lowest validation in 50 epochs and stops at 150. With 0.02 it needs 470. The bigger rate reaches the bottom sooner, but it also turns up sooner. That is why validation is checked often, not only at the end.

More tasks

Six neurons. Run the main loop with 6 hidden neurons. Does validation turn up earlier?
Without the weight. Remove the weight from the features (2 inputs instead of 3) and run the main loop. What happens to validation at epoch 1000?

Test

Check yourself

1. What is one epoch?One batch of examplesOne pass over all the training examplesOne step down the slope

2. The training error falls the whole time, while validation is lowest at epoch 200 and then grows. Which network do you keep?It does not matter: both learned from the same dataThe one from epoch 200: it predicts orders it has not seen bestThe one from the last epoch: it has the smallest error

3. After the second epoch the error is over a million, and then it becomes NaN. What is most likely?The batch is too smallThe learning rate is too big and every step overshoots furtherThe seeded generator gives bad numbers

4. All the weights are 0 at the start. What happens to the four hidden neurons?The program crashes with a division by zeroNothing special: training separates them after a few epochsThey stay identical for the whole training: the network is like one with a single hidden neuron

5. Why do you usually train with batches, not with all the examples at once or one at a time?The step is more frequent than with all of them and less noisy than with oneBecause then the network cannot memoriseBecause with batches the error never goes up

Training from start to finish

Cookies