Skip to content
wannacode

The neuron

In short: A neuron is a sum of its inputs multiplied by weights, plus a bias, passed through an activation such as the sigmoid or ReLU.

Problem

The network stopped the payment. Why?

The shop replaces its old fraud rule with a "neural network" from an outside company. On the first day a customer pays €230 from a new account with a foreign card, and the payment stops for a manual check. The customer writes to support: "Why?"

Nobody knows. For the team the network is a black box: the features of the payment go in, a number comes out. And as long as you do not know what happens inside, you cannot explain the decision, fix a mistake or even say whether the number makes sense.

The box opens. There is nothing inside you have not seen: multiplication, addition and one function.

The idea

A neuron: a sum with weights

An artificial neuron gets a few numbers (its inputs). Each input has its own weight: how strongly it matters and in which direction. The neuron multiplies each input by its weight, adds them up and adds one more number, the bias. The result is called z.

Then z goes through a function that decides what comes out. It is called the activation function. For now it is the sigmoid: it squeezes any z into a number between 0 and 1.

Sounds familiar? It is logistic regression: a weighted sum, a sigmoid, a probability and a threshold. One neuron with a sigmoid is exactly that. A neural network is many such neurons, connected one after another.

See it

The neuron in code

The three functions: the weighted sum, the sigmoid and the neuron that joins them.

static double weightedSum(double[] x, double[] w, double b) {
    double z = b; // the bias
    for (int i = 0; i < x.length; i++) {
        z += x[i] * w[i]; // input × weight
    }
    return z;
}


static double sigmoid(double z) {
    return 1 / (1 + Math.exp(-z));
}


static double neuron(double[] x, double[] w, double b) {
    return sigmoid(weightedSum(x, w, b));
}

The payments have three features: the amount in hundreds of euros, a new account (1 or 0) and a foreign card (1 or 0). The weights are set by hand so you can do the maths in your head. How a network finds them by itself is the lesson on backpropagation.

// amount (hundreds of €), new account, foreign card
double[] w = {0.9, 1.4, 1.8};
double b = -4.0;
double[][] payments = {
    {0.40, 0, 0},
    {1.20, 1, 0},
    {8.90, 1, 1},
    {0.60, 0, 1},
    {6.40, 0, 0},
    {2.30, 1, 1},
};
System.out.println("amount €   new foreign      z      p");
for (double[] x : payments) {
    double z = weightedSum(x, w, b);
    double p = sigmoid(z);
    String verdict = p >= 0.5 ? "check" : "ok";
    System.out.printf("%8.0f %5.0f %7.0f %6.2f %6.3f  %s%n",
            x[0] * 100, x[1], x[2], z, p, verdict);
}
Output
amount €   new foreign      z      p
      40     0       0  -3.64  0.026  ok
     120     1       0  -1.52  0.179  ok
     890     1       1   7.21  0.999  check
      60     0       1  -1.66  0.160  ok
     640     0       0   1.76  0.853  check
     230     1       1   1.27  0.781  check

Now you can answer the customer. The €230 payment is the last row: z = −4 + 2.3 × 0.9 + 1 × 1.4 + 1 × 1.8 = −4 + 2.07 + 1.4 + 1.8 = 1.27. The sigmoid gives 0.781, above the threshold of 0.5. The new account and the foreign card added 3.2 to the sum. Without them z would be −1.93, and the probability about 0.13.

See it

The signal goes through the neuron

static double weightedSum(double[] x, double[] w, double b) {
    double z = b; // the bias
    for (int i = 0; i < x.length; i++) {
        z += x[i] * w[i]; // input × weight
    }
    return z;
}


static double sigmoid(double z) {
    return 1 / (1 + Math.exp(-z));
}


static double neuron(double[] x, double[] w, double b) {
    return sigmoid(weightedSum(x, w, b));
}
Step by step, in words
  1. A payment of €230 from a new account with a foreign card. The inputs are 2.30, 1 and 1.
  2. z starts from the bias: −4.00.
  3. Each input times its weight: 2.30 × 0.9 = 2.07, 1 × 1.4 = 1.40, 1 × 1.8 = 1.80.
  4. The sum: z = −4 + 2.07 + 1.40 + 1.80 = 1.27.
  5. The sigmoid squeezes z between 0 and 1: σ(1.27) = 0.781.
  6. 0.781 is above the threshold of 0.5: the payment goes to a check.
  7. €40 from an old account with a Bulgarian card: z = −4 + 0.36 = −3.64, p = 0.026. Ok.
  8. A neuron: a sum with weights, a bias and a sigmoid. That is logistic regression.

The bias comes first, then each input adds its share. The sigmoid does not change the order: a bigger z gives a bigger p. It only squeezes it between 0 and 1, so it reads as a probability.

Try it

Find the bug

A colleague writes the weighted sum themselves. With a bias of −4 every payment comes out with a much smaller z than it should, and nothing reaches a check. Which line has the bug?

Pick the line with the bug.

Problem

Three neurons, worth one

A colleague wants the network to catch more complex rules. They add two neurons that read the payment and a third one that reads their outputs. There is no activation anywhere: each neuron returns the sum z itself.

The program computes the three neurons and, separately, one neuron whose weights are computed from theirs:

// two neurons read the payment, with no activation
double[] w1 = {0.5, 1.0, 1.5};
double b1 = -1.0;
double[] w2 = {1.0, -0.5, 0.5};
double b2 = 0.5;
// a third neuron reads their outputs
double[] v = {1.5, -0.5};
double c = 0.5;
// the same as one neuron with these weights
double[] w3 = new double[3];
for (int i = 0; i < 3; i++) {
    w3[i] = v[0] * w1[i] + v[1] * w2[i];
}
double b3 = v[0] * b1 + v[1] * b2 + c;
System.out.printf("w = [%.2f, %.2f, %.2f], b = %.2f%n", w3[0], w3[1], w3[2], b3);


double[][] payments = {{0.40, 0, 0}, {1.20, 1, 0}, {8.90, 1, 1}};
System.out.println("amount € three neurons   one neuron");
for (double[] x : payments) {
    double a1 = weightedSum(x, w1, b1);
    double a2 = weightedSum(x, w2, b2);
    double out = weightedSum(new double[] {a1, a2}, v, c);
    double one = weightedSum(x, w3, b3);
    System.out.printf("%8.0f %13.4f %12.4f%n", x[0] * 100, out, one);
}
Output
w = [0.25, 1.75, 2.00], b = -1.25
amount € three neurons   one neuron
      40       -1.1500      -1.1500
     120        0.8000       0.8000
     890        4.7250       4.7250

The columns match to the last digit. The three neurons are worth one. The reason is school algebra: a sum of sums is still a sum. Whatever you line up one after another, with no activation the result is one straight line (a plane, to be precise) with other weights.

The idea

The activation bends the line

Something new only comes when there is a function between the neurons that is not straight. The two most common:

  • Sigmoid: squeezes z between 0 and 1. It is handy for the output, when you want a probability.
  • ReLU (rectified linear unit): max(0, z). The positive passes, the negative becomes 0. It is the most common activation in the hidden neurons of today's networks.
static double relu(double z) {
    return Math.max(0, z);
}
double[] zs = {-3, -1, 0, 1, 3};
System.out.println("     z  sigmoid  ReLU");
for (double z : zs) {
    System.out.printf("%6.2f %8.4f %5.2f%n", z, sigmoid(z), relu(z));
}
Output
     z  sigmoid  ReLU
 -3.00   0.0474  0.00
 -1.00   0.2689  0.00
  0.00   0.5000  0.00
  1.00   0.7311  1.00
  3.00   0.9526  3.00

Both are bent. The sigmoid is an "S", and ReLU is a straight line with a bend at zero. A sum of bent lines is no longer straight, so neurons with an activation one after another can describe a rule that one neuron cannot. Such a rule comes in the next lesson.

Try it

Predict the output

A neuron with ReLU estimates how many days late a delivery is. The inputs are the distance in hundreds of kilometres and whether the parcel is heavy. A delay cannot be negative, hence ReLU. What will it print?

double[] w = {0.5, 1.0}; // hundreds of km, heavy parcel (1 or 0)
double b = -1.0;
double near = relu(weightedSum(new double[] {1.5, 0}, w, b));
double far = relu(weightedSum(new double[] {4.4, 0}, w, b));
double heavy = relu(weightedSum(new double[] {3.0, 1}, w, b));
System.out.printf("%.1f %.1f %.1f%n", near, far, heavy);
Answer

Try it

Fill in

Finish ReLU.

static double relu(double z) {
    return (0, z);
}

The idea

When one neuron is enough, and when it hurts

  • The rule is almost straight. If the features tell the answer on their own (a bigger amount, more risk), one neuron, that is logistic regression, does as well as a big network. It learns faster and needs less data.
  • There is little data. A network with hundreds of weights and 200 examples memorises them. That is overfitting.
  • You have to explain the decision. The weights of one neuron can be read: "the new account added 1.4". In a network with many layers the answer to "why" is much harder, and for a stopped payment or a refused loan it is asked for.
  • The rule is known and firm. "Above €5,000 always a manual check" is an if, not a model.
  • The inputs are on different scales. An amount in euros (40, 890) next to features of 0 and 1 gives weights that are hard to learn. That is why the amount is in hundreds of euros here. In a real model the inputs are normalised.
More precisely, if you are curious

The formulas. For a neuron with inputs x₁, x₂, x₃:

z = w₁·x₁ + w₂·x₂ + w₃·x₃ + b

p = σ(z) = 1 / (1 + e^(−z))

In words: each input times its weight, plus the bias, then the sigmoid. e is Euler's number (about 2.718), and Math.exp(-z) is e to the power −z.

Why three neurons with no activation are one. If a₁ = w₁·x + b₁ and a₂ = w₂·x + b₂, then v₁·a₁ + v₂·a₂ + c = (v₁·w₁ + v₂·w₂)·x + (v₁·b₁ + v₂·b₂ + c). That is one neuron with weights v₁·w₁ + v₂·w₂. This is exactly what the program with the three neurons computes.

Other activations: tanh (like the sigmoid, but between −1 and 1), leaky ReLU (the negative is not 0 but small, for example 0.01·z), softmax (at the output, when there are more than two classes: it turns several numbers into probabilities that add up to 1). There are many more, but ReLU and the sigmoid are enough to understand everything in this module.

What real networks are built with. Here everything is loops and arrays, so you see every number. Real networks have millions of weights, and the maths is done with matrices on a graphics card. Machine learning practice is mostly in Python: NumPy for arrays, scikit-learn for classic models, PyTorch for neural networks. Other languages have tools too: ML.NET for C#, DJL and Tribuo for Java, TensorFlow.js and ONNX Runtime for JavaScript. Often a model is trained in Python and used from a program in another language (for example through ONNX).

The name. "Neuron" comes from biology: a cell that collects signals and "fires" once their sum passes a threshold. The likeness stops there. An artificial neuron is a function of numbers.

Exercise

A neuron for reviews

Solve it in your editor.

The shop wants to sort reviews into good and bad. For each review the good words ("fast", "quality") and the bad words ("broken", "late") are already counted. Write a neuron with a sigmoid, with a weight of 1.1 for the good words, −1.7 for the bad ones and a bias of 0.2. Run it on the five reviews (good, bad): (3, 0), (1, 2), (0, 0), (2, 1), (0, 3). A review is good if p is at least 0.5.

Output
good words 3, bad words 0: p = 0.971 → good
good words 1, bad words 2: p = 0.109 → bad
good words 0, bad words 0: p = 0.550 → good
good words 2, bad words 1: p = 0.668 → good
good words 0, bad words 3: p = 0.007 → bad

The review with no words at all comes out good with 0.550: that is the bias alone. With nothing else to go on, the model slightly believes the review is good.

More tasks

  • A third feature. Add the exclamation marks as a feature with a weight of −0.3 and see which reviews change their decision.
  • The threshold. Use a threshold of 0.7 instead of 0.5. Which reviews become bad?

Test

Check yourself

1. What does one neuron compute?
2. A neuron with a sigmoid and a threshold of 0.5 on the features of a payment. What is it?
3. A colleague puts three neurons one after another, with no activation, to catch more complex rules. What do they get?
4. Why does ReLU return 0 for a negative z?
5. When is one neuron (or logistic regression) a better choice than a big network?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·