Skip to content
wannacode

Layers

In short: Layers connect neurons one after another: a hidden layer lets the network learn rules that one straight line cannot draw, such as XOR.

Problem

The rule one line cannot draw

The fraud team wants a new rule: a payment is suspicious when the country of the card and the country of the connection (by IP address) differ. A Bulgarian card from a foreign IP: someone is using a stolen card abroad. A foreign card from a Bulgarian IP: a stolen card used here. A Bulgarian card from Bulgaria and a foreign card from abroad (a customer who lives there) are fine.

There are two features, 0 or 1 each, where 1 means abroad. There are four cases, and (1, 0) and (0, 1) are suspicious. The program tries 15,000 neurons: every weight from −6 to 6 in steps of 0.5 and a bias from −5.75 to 5.75. For each it counts how many cases it gets right at a threshold of 0.5:

const cases = [[0, 0], [1, 0], [0, 1], [1, 1]];
const labels = [0, 1, 1, 0]; // 1 = suspicious
let tried = 0;
let best = 0;
let bestW = [0, 0];
let bestB = 0;
for (let i = -12; i <= 12; i++) {
  for (let j = -12; j <= 12; j++) {
    for (let k = -12; k <= 11; k++) {
      const w = [i / 2, j / 2];
      const b = k / 2 + 0.25;
      tried++;
      let right = 0;
      cases.forEach((x, n) => {
        const guess = neuron(x, w, b) >= 0.5 ? 1 : 0;
        if (guess === labels[n]) right++;
      });
      if (right > best) {
        best = right;
        bestW = w;
        bestB = b;
      }
    }
  }
}
console.log(`neurons tried: ${tried}, the best: ${best} of 4`);
console.log(`w = [${bestW[0].toFixed(2)}, ${bestW[1].toFixed(2)}], b = ${bestB.toFixed(2)}`);
cases.forEach((x, n) => {
  const p = neuron(x, bestW, bestB);
  const right = (p >= 0.5 ? 1 : 0) === labels[n];
  console.log(`${x[0]} ${x[1]}: p = ${p.toFixed(3)} ${right ? "right" : "wrong"}`);
});
Output
neurons tried: 15000, the best: 3 of 4
w = [-6.00, 0.50], b = -0.25
0 0: p = 0.438 right
1 0: p = 0.002 wrong
0 1: p = 0.562 right
1 1: p = 0.003 right

The best gets 3 of 4, and it calls a foreign card from a Bulgarian IP "ok". None of the 15,000 gets all four. And it is not because the grid of weights is too coarse: the next card shows why.

The idea

Why one line is not enough

Draw the cases as points on a square: the card is the horizontal axis, the connection is the vertical one. The suspicious (1, 0) and (0, 1) are in two opposite corners, and the other two are fine.

The neuron says "yes" when z is above zero. The boundary z = 0, that is w₁·card + w₂·IP + b = 0, is a straight line. On one side of it the neuron says "yes", on the other "no". And a line that leaves the two suspicious corners on one side leaves at least one of the other two with them. Try it with a pencil: it does not work.

The logistic regression from the classification lesson has the same boundary, so it cannot do it either. Two lines are needed.

The idea

Two neurons in the middle

Between the input and the output we put two neurons. Each draws its own line:

  • h1: "at least one is from abroad". Weights 6 and 6, bias −3: z is above zero as soon as at least one input is 1.
  • h2: "both are from abroad". Weights 6 and 6, bias −9: z is above zero only when both inputs are 1.

The output reads h1 and h2 with weights 10 and −10 and a bias of −5: "at least one, but not both". The suspicious area is the strip between the two lines.

The neurons in the middle are a hidden layer. Hidden, because nobody gives them a right answer: the input comes from the data, the output has a label, and what the ones in between compute, the network decides itself. Here the weights are chosen by hand so that they mean something you can say in words. Learned ones usually cannot be put in words that easily.

Computing from left to right, layer by layer, is the forward pass:

const W = [[6, 6], [6, 6]];
const B = [-3, -9];
const V = [10, -10];
const C = -5;


function forward(x) {
  // at least one is from abroad
  const h1 = neuron(x, W[0], B[0]);
  // both are from abroad
  const h2 = neuron(x, W[1], B[1]);
  const h = [h1, h2]; // the hidden layer
  return neuron(h, V, C);
}

See it

The forward pass, number by number

One case: a foreign card from a Bulgarian IP, (1, 0). Each step prints its z and its output:

const x = [1, 0]; // a foreign card, an IP from Bulgaria
const z1 = weightedSum(x, W[0], B[0]);
const h1 = sigmoid(z1);
console.log(`h1: z = ${z1.toFixed(2)}, h1 = ${h1.toFixed(4)}`);
const z2 = weightedSum(x, W[1], B[1]);
const h2 = sigmoid(z2);
console.log(`h2: z = ${z2.toFixed(2)}, h2 = ${h2.toFixed(4)}`);
const h = [h1, h2];
const z = weightedSum(h, V, C);
const p = sigmoid(z);
console.log(`output: z = ${z.toFixed(2)}, p = ${p.toFixed(4)}`);
Output
h1: z = 3.00, h1 = 0.9526
h2: z = -3.00, h2 = 0.0474
output: z = 4.05, p = 0.9829

h1: 6 × 1 + 6 × 0 − 3 = 3, and the sigmoid gives 0.9526. h2: 6 × 1 + 6 × 0 − 9 = −3, that is 0.0474. The output: 10 × 0.9526 − 10 × 0.0474 − 5 = 4.05, and the sigmoid gives 0.9829. Suspicious.

The four cases with forward:

const cases = [[0, 0], [1, 0], [0, 1], [1, 1]];
const fmt = (v, d, n = 0) => v.toFixed(d).padStart(n);
console.log(" card  IP       p");
for (const x of cases) {
  const p = forward(x);
  const verdict = p >= 0.5 ? "suspicious" : "ok";
  console.log(`${fmt(x[0], 0, 5)} ${fmt(x[1], 0, 3)} ${fmt(p, 4, 7)}  ${verdict}`);
}
Output
 card  IP       p
    0   0  0.0107  ok
    1   0  0.9829  suspicious
    0   1  0.9829  suspicious
    1   1  0.0107  ok

Four out of four. The same weights, the same sigmoid, just one more layer.

See it

Two lines and the signal through the network

const W = [[6, 6], [6, 6]];
const B = [-3, -9];
const V = [10, -10];
const C = -5;


function forward(x) {
  // at least one is from abroad
  const h1 = neuron(x, W[0], B[0]);
  // both are from abroad
  const h2 = neuron(x, W[1], B[1]);
  const h = [h1, h2]; // the hidden layer
  return neuron(h, V, C);
}
Step by step, in words
  1. Four cases: card and IP, 1 means abroad. (1, 0) and (0, 1) are suspicious.
  2. The best single neuron draws one line and gets 3 of 4: (1, 0) stays on the wrong side.
  3. The hidden layer: two lines, h1 "at least one" and h2 "both". Suspicious is between them.
  4. The forward pass for a foreign card from a Bulgarian IP: x = (1, 0).
  5. h1: 6 × 1 + 6 × 0 − 3 = 3.00, σ gives 0.9526. h2: 6 × 1 + 6 × 0 − 9 = −3.00, σ gives 0.0474.
  6. The output reads h: 10 × 0.9526 − 10 × 0.0474 − 5 = 4.05, σ gives 0.9829.
  7. 0.9829 is above 0.5: suspicious.
  8. The four cases through the same network: 0.0107, 0.9829, 0.9829, 0.0107. Four of four.

At the top are the four cases. One line always leaves one wrong. The two lines of the hidden layer close a strip, and the output says "suspicious" only inside it.

Try it

Predict the output

The same rule with ReLU instead of the sigmoid: h1 = ReLU(card + IP), h2 = ReLU(card + IP − 1), output h1 − 2·h2. What will it print for the four cases?

const cases = [[0, 0], [1, 0], [0, 1], [1, 1]];
const out = cases.map((x) => {
  const h1 = relu(x[0] + x[1]);
  const h2 = relu(x[0] + x[1] - 1);
  return h1 - 2 * h2;
});
console.log(out.join(" "));
Answer

Try it

Find the bug

A colleague writes forward themselves. The network answers like one neuron: three out of four. Which line has the bug?

Pick the line with the bug.

Try it

Fill in

What does the output neuron get?

const h = [h1, h2];
return neuron(, V, C);

The idea

When a hidden layer helps, and when it hurts

  • It helps when the features affect each other. "Foreign card" alone means nothing, and neither does "foreign IP". The combination does. A hidden layer learns such combinations without being told.
  • If you know the combination, write it. The feature "card ≠ connection" is one line of code. With it one neuron or an if is enough. A network that learns something you could write is slower, needs data and makes more mistakes.
  • More neurons are more weights. Here there are 9 numbers. A hidden layer of 100 neurons over 20 features is 2,201 numbers. More weights means more data, or the network memorises.
  • Every layer costs on every payment. The forward pass goes through all the weights for every request. With a big network and many payments that is time and money.
  • Hidden neurons do not have to mean anything. Here they do, because the weights are chosen by hand. Learned ones are usually a mix that cannot be put into words.
More precisely, if you are curious

With formulas. The hidden layer is h = σ(W·x + b): W is a table (a matrix) of 2 × 2 weights, b holds the two biases, and σ is applied to each element. The output is p = σ(v·h + c). The forward pass is these two lines one after another. That is why in real libraries a layer is a matrix times a vector.

Deep learning. A network with many hidden layers is called deep. Each layer builds on the one before: the first ones catch simple combinations, the next ones combine those.

How many neurons are enough. It has been proved that one hidden layer with enough neurons can approximate any continuous function (on a bounded region) as closely as you like. But "enough" may be a lot, and nobody says how to find the weights. That is why in practice there are several layers.

A bit of history. In 1969 Minsky and Papert showed that a single layer (a perceptron) cannot learn exactly this rule (XOR, "one or the other"). Interest in neural networks dropped for years. The fix with a hidden layer was known, but not how to learn its weights. That came with backpropagation.

Exercise

One layer for all

Solve it in your editor.

The lesson's forward knows that there are two hidden neurons. Write a general layer(x, w, b): it gets the inputs, a table of weights (one row for each neuron) and the biases, and returns an array with the output of each neuron. Then build the same network as two layers: the hidden one with the weights from the lesson and the output one with a single neuron (weights 10 and −10, bias −5). Run the four cases through layer(layer(x, w1, b1), w2, b2).

Output
 card  IP       p
    0   0  0.0107  ok
    1   0  0.9829  suspicious
    0   1  0.9829  suspicious
    1   1  0.0107  ok

The same table as in the lesson: layer is the general part, and the network is two lines of weights.

More tasks

  • A third hidden neuron. Add a neuron with weights 0 and 0 and a bias of 0 to the hidden layer (and a weight of 0 for it at the output). The output must not change.
  • ReLU in the hidden layer. Give layer the activation as a parameter and build the network from "Predict the output".

Test

Check yourself

1. Why can one neuron not learn "the card and the connection are from different countries"?
2. What is a hidden layer?
3. What is the forward pass?
4. The hidden layer has two neurons with a sigmoid, and the output is a sigmoid. How many numbers does the network learn?
5. You know that what is suspicious is the two countries differing. What is most sensible?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·