Layers
In short: Layers connect neurons one after another: a hidden layer lets the network learn rules that one straight line cannot draw, such as XOR.
Problem
The rule one line cannot draw
The fraud team wants a new rule: a payment is suspicious when the country of the card and the country of the connection (by IP address) differ. A Bulgarian card from a foreign IP: someone is using a stolen card abroad. A foreign card from a Bulgarian IP: a stolen card used here. A Bulgarian card from Bulgaria and a foreign card from abroad (a customer who lives there) are fine.
There are two features, 0 or 1 each, where 1 means abroad. There are four cases, and (1, 0) and (0, 1) are suspicious. The program tries 15,000 neurons: every weight from −6 to 6 in steps of 0.5 and a bias from −5.75 to 5.75. For each it counts how many cases it gets right at a threshold of 0.5:
cases = [[0, 0], [1, 0], [0, 1], [1, 1]]labels = [0, 1, 1, 0] # 1 = suspicioustried, best = 0, 0best_w, best_b = [0.0, 0.0], 0.0for i in range(-12, 13): for j in range(-12, 13): for k in range(-12, 12): w = [i / 2, j / 2] b = k / 2 + 0.25 tried += 1 right = 0 for x, label in zip(cases, labels): guess = 1 if neuron(x, w, b) >= 0.5 else 0 if guess == label: right += 1 if right > best: best, best_w, best_b = right, w, bprint(f"neurons tried: {tried}, the best: {best} of 4")print(f"w = [{best_w[0]:.2f}, {best_w[1]:.2f}], b = {best_b:.2f}")for x, label in zip(cases, labels): p = neuron(x, best_w, best_b) right = (1 if p >= 0.5 else 0) == label print(f"{x[0]} {x[1]}: p = {p:.3f} {'right' if right else 'wrong'}")neurons tried: 15000, the best: 3 of 4w = [-6.00, 0.50], b = -0.250 0: p = 0.438 right1 0: p = 0.002 wrong0 1: p = 0.562 right1 1: p = 0.003 rightThe best gets 3 of 4, and it calls a foreign card from a Bulgarian IP "ok". None of the 15,000 gets all four. And it is not because the grid of weights is too coarse: the next card shows why.
The idea
Why one line is not enough
Draw the cases as points on a square: the card is the horizontal axis, the connection is the vertical one. The suspicious (1, 0) and (0, 1) are in two opposite corners, and the other two are fine.
The neuron says "yes" when z is above zero. The boundary z = 0, that is w₁·card + w₂·IP + b = 0, is a straight line. On one side of it the neuron says "yes", on the other "no". And a line that leaves the two suspicious corners on one side leaves at least one of the other two with them. Try it with a pencil: it does not work.
The logistic regression from the classification lesson has the same boundary, so it cannot do it either. Two lines are needed.
The idea
Two neurons in the middle
Between the input and the output we put two neurons. Each draws its own line:
- h1: "at least one is from abroad". Weights 6 and 6, bias −3: z is above zero as soon as at least one input is 1.
- h2: "both are from abroad". Weights 6 and 6, bias −9: z is above zero only when both inputs are 1.
The output reads h1 and h2 with weights 10 and −10 and a bias of −5: "at least one, but not both". The suspicious area is the strip between the two lines.
The neurons in the middle are a hidden layer. Hidden, because nobody gives them a right answer: the input comes from the data, the output has a label, and what the ones in between compute, the network decides itself. Here the weights are chosen by hand so that they mean something you can say in words. Learned ones usually cannot be put in words that easily.
Computing from left to right, layer by layer, is the forward pass:
W = [[6.0, 6.0], [6.0, 6.0]]B = [-3.0, -9.0]V = [10.0, -10.0]C = -5.0
def forward(x): # at least one is from abroad h1 = neuron(x, W[0], B[0]) # both are from abroad h2 = neuron(x, W[1], B[1]) h = [h1, h2] # the hidden layer return neuron(h, V, C)forward reads them directly.See it
The forward pass, number by number
One case: a foreign card from a Bulgarian IP, (1, 0). Each step prints its z and its output:
x = [1, 0] # a foreign card, an IP from Bulgariaz1 = weighted_sum(x, W[0], B[0])h1 = sigmoid(z1)print(f"h1: z = {z1:.2f}, h1 = {h1:.4f}")z2 = weighted_sum(x, W[1], B[1])h2 = sigmoid(z2)print(f"h2: z = {z2:.2f}, h2 = {h2:.4f}")h = [h1, h2]z = weighted_sum(h, V, C)p = sigmoid(z)print(f"output: z = {z:.2f}, p = {p:.4f}")h1: z = 3.00, h1 = 0.9526h2: z = -3.00, h2 = 0.0474output: z = 4.05, p = 0.9829h1: 6 × 1 + 6 × 0 − 3 = 3, and the sigmoid gives 0.9526. h2: 6 × 1 + 6 × 0 − 9 = −3, that is 0.0474. The output: 10 × 0.9526 − 10 × 0.0474 − 5 = 4.05, and the sigmoid gives 0.9829. Suspicious.
The four cases with forward:
cases = [[0, 0], [1, 0], [0, 1], [1, 1]]print(" card IP p")for x in cases: p = forward(x) verdict = "suspicious" if p >= 0.5 else "ok" print(f"{x[0]:5.0f} {x[1]:3.0f} {p:7.4f} {verdict}") card IP p 0 0 0.0107 ok 1 0 0.9829 suspicious 0 1 0.9829 suspicious 1 1 0.0107 okFour out of four. The same weights, the same sigmoid, just one more layer.
See it
Two lines and the signal through the network
W = [[6.0, 6.0], [6.0, 6.0]]B = [-3.0, -9.0]V = [10.0, -10.0]C = -5.0
def forward(x): # at least one is from abroad h1 = neuron(x, W[0], B[0]) # both are from abroad h2 = neuron(x, W[1], B[1]) h = [h1, h2] # the hidden layer return neuron(h, V, C)Step by step, in words
- Four cases: card and IP, 1 means abroad. (1, 0) and (0, 1) are suspicious.
- The best single neuron draws one line and gets 3 of 4: (1, 0) stays on the wrong side.
- The hidden layer: two lines, h1 "at least one" and h2 "both". Suspicious is between them.
- The forward pass for a foreign card from a Bulgarian IP: x = (1, 0).
- h1: 6 × 1 + 6 × 0 − 3 = 3.00, σ gives 0.9526. h2: 6 × 1 + 6 × 0 − 9 = −3.00, σ gives 0.0474.
- The output reads h: 10 × 0.9526 − 10 × 0.0474 − 5 = 4.05, σ gives 0.9829.
- 0.9829 is above 0.5: suspicious.
- The four cases through the same network: 0.0107, 0.9829, 0.9829, 0.0107. Four of four.
At the top are the four cases. One line always leaves one wrong. The two lines of the hidden layer close a strip, and the output says "suspicious" only inside it.
Try it
Predict the output
The same rule with ReLU instead of the sigmoid: h1 = ReLU(card + IP), h2 = ReLU(card + IP − 1), output h1 − 2·h2. What will it print for the four cases?
cases = [[0, 0], [1, 0], [0, 1], [1, 1]]out = []for x in cases: h1 = relu(x[0] + x[1]) h2 = relu(x[0] + x[1] - 1) out.append(h1 - 2 * h2)print(" ".join(f"{o:.0f}" for o in out))(0, 0): 0 − 0 = 0. (1, 0) and (0, 1): h1 = 1, h2 = ReLU(0) = 0, the output is 1. (1, 1): h1 = 2, h2 = 1, the output is 2 − 2 = 0. With ReLU the rule comes out exactly, with no rounding. Without h2 the last number would be 2.
Try it
Find the bug
A colleague writes forward themselves. The network answers like one neuron: three out of four. Which line has the bug?
Pick the line with the bug.
The output reads the input x, not the hidden layer h. The hidden neurons are computed and not used, and the network collapses into one neuron with its one line. The output has to get h.
Try it
Fill in
What does the output neuron get?
h = [h1, h2]
return neuron(, V, C)The output reads h: the outputs of the hidden layer. Each layer gets the outputs of the one before.
The idea
When a hidden layer helps, and when it hurts
- It helps when the features affect each other. "Foreign card" alone means nothing, and neither does "foreign IP". The combination does. A hidden layer learns such combinations without being told.
- If you know the combination, write it. The feature "card ≠ connection" is one line of code. With it one neuron or an
ifis enough. A network that learns something you could write is slower, needs data and makes more mistakes. - More neurons are more weights. Here there are 9 numbers. A hidden layer of 100 neurons over 20 features is 2,201 numbers. More weights means more data, or the network memorises.
- Every layer costs on every payment. The forward pass goes through all the weights for every request. With a big network and many payments that is time and money.
- Hidden neurons do not have to mean anything. Here they do, because the weights are chosen by hand. Learned ones are usually a mix that cannot be put into words.
More precisely, if you are curious
With formulas. The hidden layer is h = σ(W·x + b): W is a table (a matrix) of 2 × 2 weights, b holds the two biases, and σ is applied to each element. The output is p = σ(v·h + c). The forward pass is these two lines one after another. That is why in real libraries a layer is a matrix times a vector.
Deep learning. A network with many hidden layers is called deep. Each layer builds on the one before: the first ones catch simple combinations, the next ones combine those.
How many neurons are enough. It has been proved that one hidden layer with enough neurons can approximate any continuous function (on a bounded region) as closely as you like. But "enough" may be a lot, and nobody says how to find the weights. That is why in practice there are several layers.
A bit of history. In 1969 Minsky and Papert showed that a single layer (a perceptron) cannot learn exactly this rule (XOR, "one or the other"). Interest in neural networks dropped for years. The fix with a hidden layer was known, but not how to learn its weights. That came with backpropagation.
Exercise
One layer for all
Solve it in your editor.
The lesson's forward knows that there are two hidden neurons. Write a general layer(x, w, b): it gets the inputs, a table of weights (one row for each neuron) and the biases, and returns an array with the output of each neuron. Then build the same network as two layers: the hidden one with the weights from the lesson and the output one with a single neuron (weights 10 and −10, bias −5). Run the four cases through layer(layer(x, w1, b1), w2, b2).
card IP p 0 0 0.0107 ok 1 0 0.9829 suspicious 0 1 0.9829 suspicious 1 1 0.0107 okThe same table as in the lesson: layer is the general part, and the network is two lines of weights.
More tasks
- A third hidden neuron. Add a neuron with weights 0 and 0 and a bias of 0 to the hidden layer (and a weight of 0 for it at the output). The output must not change.
- ReLU in the hidden layer. Give
layerthe activation as a parameter and build the network from "Predict the output".
Test