Skip to content
wannacode

Classification: fraud or not

In short: Binary classification answers “yes” or “no”: logistic regression squeezes a number with the sigmoid into a probability between 0 and 1, and a threshold decides.

Problem

Fraud or not

The customer presses “Pay”. The shop has a few milliseconds to decide: does it let the payment through, or block it for a check? If it lets a fraud through, it loses the goods and pays a fee for the reversed payment. If it blocks an honest customer, the customer gets angry and buys somewhere else.

Here the answer is not a number like the delivery time, but “fraud” or “not”. We have 20 past payments. The feature is the amount compared with the customer’s usual one: 3.0 means “three times more than usual”. The label is 1 for fraud and 0 for an honest payment:

// the amount compared with the customer’s usual: 3.0 is three times more
double[] ratios = [0.4, 0.6, 0.7, 0.9, 1.0, 1.1, 1.2, 1.3, 1.5, 2.0, 2.6, 3.2,
    0.8, 3.6, 4.1, 5.0, 6.0, 7.5, 8.2, 9.0];
// 1 = fraud, 0 = honest payment
int[] fraud = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    1, 1, 1, 1, 1, 1, 1, 1];

The first idea is the line from linear regression: the labels are 0 and 1, and above 0.5 we block. Learned with the same descent:

var line = TrainLine(ratios, fraud, 0.01, 5000);
Console.WriteLine($"line, payment 0.4×: {line.W * 0.4 + line.B:F2}");
Console.WriteLine($"line, payment 9.0×: {line.W * 9.0 + line.B:F2}");
// a small step: 40× is a big number
double[] ratios2 = [.. ratios, 40.0];
int[] fraud2 = [.. fraud, 1];
var moved = TrainLine(ratios2, fraud2, 0.01, 5000);
Report("line", line, ratios2, fraud2);
Report("line with the 40× payment", moved, ratios2, fraud2);
Output
line, payment 0.4×: 0.03
line, payment 9.0×: 1.24
line: 0.5 at 3.75×, missed frauds: 2 of 9
line with the 40× payment: 0.5 at 7.31×, missed frauds: 5 of 9

The line says 1.24 for a 9× payment. That is not a probability: “124% fraud” means nothing. The worse part is below. One fraud arrives at 40× the usual amount. It is obvious, and yet it tilts the line so much that its boundary moves from 3.75× to 7.31×. Now the line misses 5 of 9 frauds instead of 2. One clear fraud made the model worse for all the others.

The idea

The answer is a class

When the model picks between a few answers (“fraud” or “honest”, a “bad” or “good” review), the task is called classification, and the answers are classes. In linear regression the answer was a number on a straight line. Here we want something else: a number between 0 and 1 that says how likely the fraud is. Then we decide with it.

We need two things:

  • a function that squeezes any number between 0 and 1 without cutting it;
  • a boundary above which we block the payment.

The idea

The sigmoid

The sigmoid is sigmoid(z) = 1 / (1 + e−z). A big z makes it almost 1, a big negative z makes it almost 0, and z = 0 gives exactly 0.5:

static double Sigmoid(double z) =>
    1 / (1 + Math.Exp(-z));
for (var z = -4; z <= 4; z += 2)
{
    Console.WriteLine($"z = {z}: {Sigmoid(z):F2}");
}
Output
z = -4: 0.02
z = -2: 0.12
z = 0: 0.50
z = 2: 0.88
z = 4: 0.98

However big z is, the result never goes past 1. That is why the 40× payment cannot “pull” the curve like it pulled the line: for it the sigmoid already says almost 1, and there is nothing more to fix.

See it

Logistic regression

The model is the same line, but passed through the sigmoid. First it computes a score: w · x + b, as in linear regression. Then the sigmoid turns it into a probability p. Finally the threshold decides: p ≥ 0.5 means “block”.

static double Sigmoid(double z) =>
    1 / (1 + Math.Exp(-z));


static bool Block(double ratio, double w, double b,
                  double threshold)
{
    var score = w * ratio + b;
    var p = Sigmoid(score);
    return p >= threshold;
}

This is logistic regression. The name is misleading: it does not predict a number, it predicts a class. “Regression” is because of the line inside.

With the learned w and b (how, is in the next card), for three new payments:

var (w, b) = TrainLogistic(ratios, fraud, 0.1, 3000);
foreach (var ratio in new[] { 1.0, 3.0, 5.0 })
{
    var p = Sigmoid(w * ratio + b);
    var verdict = Block(ratio, w, b, 0.5) ? "block" : "allow";
    Console.WriteLine($"{ratio:F1}×: probability of fraud {p:F2} → {verdict}");
}
Output
1.0×: probability of fraud 0.06 → allow
3.0×: probability of fraud 0.46 → allow
5.0×: probability of fraud 0.91 → block

A 3× payment is 0.46: suspicious, but below the threshold. The number between 0 and 1 is useful on its own. The shop can let 0.46 through without questions, ask the bank for a confirmation (3-D Secure) at 0.6, and block 0.95.

See it

Learned with the same descent

How are w and b found? Exactly as in the gradient descent lesson: we start from zero, compute how wrong the model is for every payment, and move the weights a little against the slope. Here is the training of the line from the problem, and next to it the logistic one:

static (double W, double B) TrainLine(double[] xs, int[] ys, double rate, int epochs)
{
    double w = 0, b = 0;
    int n = xs.Length;
    for (var epoch = 0; epoch < epochs; epoch++)
    {
        double gw = 0, gb = 0;
        for (var i = 0; i < n; i++)
        {
            var p = w * xs[i] + b;
            gw += (p - ys[i]) * xs[i];
            gb += p - ys[i];
        }
        w -= rate * gw / n;
        b -= rate * gb / n;
    }
    return (w, b);
}
static (double W, double B) TrainLogistic(double[] xs, int[] ys, double rate, int epochs)
{
    double w = 0, b = 0;
    int n = xs.Length;
    for (var epoch = 0; epoch < epochs; epoch++)
    {
        double gw = 0, gb = 0;
        for (var i = 0; i < n; i++)
        {
            var p = Sigmoid(w * xs[i] + b); // the only line that differs
            gw += (p - ys[i]) * xs[i];
            gb += p - ys[i];
        }
        w -= rate * gw / n;
        b -= rate * gb / n;
    }
    return (w, b);
}

Only one line is different: the prediction goes through the sigmoid. The step (p − y) · x is the same. If a fraud got 0.9, the error is small and the weights barely move. If it got 0.1, the error is big and so is the step.

To see that a model learns, we measure its error. For “yes” and “no” the measure is log loss: how surprised the model is by the right answer. Why not the mean squared error is in “More” at the end.

static double LogLoss(double[] xs, int[] ys, double w, double b)
{
    double sum = 0;
    for (var i = 0; i < xs.Length; i++)
    {
        var p = Sigmoid(w * xs[i] + b);
        sum -= ys[i] == 1 ? Math.Log(p) : Math.Log(1 - p);
    }
    return sum / xs.Length;
}
foreach (var epochs in new[] { 10, 100, 1000, 3000 })
{
    var (w, b) = TrainLogistic(ratios, fraud, 0.1, epochs);
    var loss = LogLoss(ratios, fraud, w, b);
    Console.WriteLine($"epoch {epochs}: loss {loss:F4}, w = {w:F4}, b = {b:F4}");
}
Output
epoch 10: loss 0.5895, w = 0.2139, b = -0.1837
epoch 100: loss 0.3899, w = 0.5246, b = -1.4580
epoch 1000: loss 0.2906, w = 1.1696, b = -3.6490
epoch 3000: loss 0.2897, w = 1.2624, b = -3.9459

The loss falls and stops around 0.29. The weights reach w = 1.26 and b = −3.95. The sigmoid passes 0.5 where the score is 0, that is at x = −b / w ≈ 3.13×. And the 40× payment from the problem?

double[] ratios2 = [.. ratios, 40.0];
int[] fraud2 = [.. fraud, 1];
Report("sigmoid", TrainLogistic(ratios, fraud, 0.1, 3000), ratios2, fraud2);
Report("sigmoid with the 40× payment", TrainLogistic(ratios2, fraud2, 0.1, 3000), ratios2, fraud2);
Output
sigmoid: 0.5 at 3.13×, missed frauds: 1 of 9
sigmoid with the 40× payment: 0.5 at 3.13×, missed frauds: 1 of 9

The boundary does not move. The only missed fraud is still the one at 0.8×: a small payment with which the thief checks that the card works. By its amount it looks like the honest ones, and no threshold on this amount will catch it. It needs another feature (a new country, a new device, night time).

See it

The sigmoid and the threshold

static double Sigmoid(double z) =>
    1 / (1 + Math.Exp(-z));


static bool Block(double ratio, double w, double b,
                  double threshold)
{
    var score = w * ratio + b;
    var p = Sigmoid(score);
    return p >= threshold;
}
Step by step, in words
  1. The 20 payments: across is the amount compared with the customer’s usual. Frauds are at the top (1), honest ones at the bottom (0).
  2. A straight line through the zeros and ones: at 9× it says 1.24. That is not a probability.
  3. The sigmoid bends the line between 0 and 1. It crosses 0.5 where the score is 0: at 3.13×.
  4. Every payment gets its probability and sits on the curve. Red ones are frauds, green ones honest.
  5. Threshold 0.5: above it we block, below it we let it through. Missed frauds: 1, honest customers blocked: 1.
  6. Threshold 0.3 is stricter: missed frauds: 1, honest customers blocked: 2.
  7. Threshold 0.7 is softer: missed frauds: 2, honest customers blocked: 0.
  8. The model gives a probability, and the threshold is a decision: a lower one catches more fraud and blocks more honest customers.

The line goes above 1 and below 0. The sigmoid stays between them, and the horizontal threshold line cuts the payments into “block” and “allow”. When the threshold goes down, fewer frauds get through, but more honest customers are blocked too.

Try it

Fill in

Which single line turns the training of the line into logistic regression?

for (var i = 0; i < n; i++)
{
    var p = (w * xs[i] + b);
    gw += (p - ys[i]) * xs[i];
    gb += p - ys[i];
}

Try it

Predict the output

The model has w = 1.26 and b = −3.95, and the threshold is 0.5. What does it print for payments of 1.5×, 3.0× and 3.5×?

double w = 1.26, b = -3.95;
foreach (var ratio in new[] { 1.5, 3.0, 3.5 })
{
    Console.WriteLine(Block(ratio, w, b, 0.5) ? "block" : "allow");
}
Answer

The idea

The threshold is a business decision

The model gives a probability. What we do with it is decided by the threshold, and the model does not learn the threshold. Here are the same 20 payments at four thresholds:

foreach (var threshold in new[] { 0.3, 0.5, 0.7, 0.9 })
{
    int missed = 0, blocked = 0;
    for (var i = 0; i < ratios.Length; i++)
    {
        var stop = Block(ratios[i], w, b, threshold);
        if (fraud[i] == 1 && !stop) missed++;
        if (fraud[i] == 0 && stop) blocked++;
    }
    Console.WriteLine($"threshold {threshold:F1}: missed frauds {missed}, honest blocked {blocked}");
}
Output
threshold 0.3: missed frauds 1, honest blocked 2
threshold 0.5: missed frauds 1, honest blocked 1
threshold 0.7: missed frauds 2, honest blocked 0
threshold 0.9: missed frauds 3, honest blocked 0

A low threshold is strict: at 0.3 we miss 1 fraud but block 2 honest customers. A high threshold is soft: at 0.9 we block no honest customer but miss 3 frauds. There is no threshold without mistakes, because frauds and honest payments overlap.

Which threshold is right? It depends on what each mistake costs. A missed fraud of €300 costs €300 plus the fee. A blocked honest customer costs one angry person, a call to support and sometimes a lost customer. How to compare the two with numbers, when frauds are 1 in 100, is the next lesson.

Try it

Find the bug

A colleague writes the sigmoid straight into block. Their tests use frauds and pass. On Friday the shop blocks every payment, the honest ones too. Which line has the bug?

Pick the line with the bug.

The idea

When it hurts

Logistic regression is a good first solution: it is fast, easy to explain (“we blocked it because the amount is 6 times the usual”) and it gives a probability. Here is when it is not enough or does harm:

  • There is one boundary. The model blocks above one threshold on a weighted sum of the features. If the dangerous part is in the middle (neither very small nor very large amounts), one boundary cannot separate it. You need another feature or another model, such as a tree or a neural network (the next module).
  • The probability is as true as the data. Here frauds are 8 of 20, because that way we have something to learn from. In the real stream they are about 1 in 100. A model trained on 40% fraud sees fraud too often. That is the next lesson.
  • A rule that is the law is not learned. “We do not accept cards from a country under sanctions” is an if, not a model. The model is for what you cannot write as a rule.
  • A feature that does not exist at decision time. “The customer disputed the payment” is a great fraud feature, but it arrives weeks after the payment. If it is in the training data, the model looks excellent, and at decision time it is blind.
  • Decisions about people. Credit, jobs, insurance: there the model has to be fair to groups of people and explainable. That is a separate topic, and often the law.
More precisely, if you are curious

Why log loss and not MSE. Log loss for one payment is −log(p) if it is fraud and −log(1 − p) if it is honest. If the model says 0.01 for a real fraud, the error is −log(0.01) ≈ 4.6. With MSE the same mistake is only (1 − 0.01)² ≈ 0.98. Log loss punishes a confident wrong answer much harder. And the slope of log loss with respect to w is exactly (p − y) · x, which is why the code is so short. With a sigmoid and MSE the slope has an extra factor p · (1 − p), which becomes almost 0 when the model is confident and wrong: learning stops exactly when it should be fastest.

Why the steps differ. The line is learned with a step of 0.01, logistic regression with 0.1. For the line the error grows with x, and the 40× payment gives a huge slope: with a step of 0.1 the line blows up. For the sigmoid p − y is between −1 and 1, so even a big x does not shake it as much.

More features. With several features (the amount, is it night, is the device new) the score is w₁ · x₁ + w₂ · x₂ + w₃ · x₃ + b, and the code is the same with arrays of weights. The boundary is a line in a plane, or a plane in space: still one.

More than two classes. For “good / neutral / bad” there is one score per class, and softmax turns them into probabilities that add up to 1. You will meet it with language models: they pick the next token exactly like that, among tens of thousands of classes.

In real code. In scikit-learn this is LogisticRegression, in ML.NET it is SdcaLogisticRegression, in Tribuo (Java) it is LogisticRegressionTrainer. Inside is the same idea with a smarter descent and with regularization, which you will see two lessons later.

:F2 writes the number according to the system culture. On a computer with a Bulgarian culture you will see 0,46, and the lesson check runs with the invariant culture and a dot. (double W, double B) is a tuple with names: the function returns two numbers at once.

Exercise

Is the review bad

Solve it in your editor.

Under every product there are reviews. The shop wants the bad ones to reach support at once. The feature is how many words from a list (“broken”, “returned”, “late”…) the review has. You have 18 reviews: the number of bad words is 0, 0, 1, 0, 2, 0, 1, 0, 1, 3, 2, 3, 4, 1, 5, 3, 2, 4, and the labels (1 = bad review) are ten zeros, then eight ones. Train logistic regression with a step of 0.5 for 2000 epochs and print the probability and the decision (threshold 0.5) for a review with 0 to 5 bad words.

Output
bad words: 0 → 0.05, good
bad words: 1 → 0.20, good
bad words: 2 → 0.52, bad
bad words: 3 → 0.83, bad
bad words: 4 → 0.96, bad
bad words: 5 → 0.99, bad

A review with 2 bad words is on the edge: 0.52. That is also where the good reviews that say “not broken, but late” are.

More tasks

  • A different threshold. Support wants to see everything suspicious. Which threshold also sends them the reviews with 1 bad word?
  • A new feature. Add the stars (1 to 5) as a second feature: w₁ · words + w₂ · stars + b. How much does the probability change for 2 bad words and 5 stars?

Test

Check yourself

1. Why is a line learned on labels 0 and 1 a poor model for fraud?
2. What does the sigmoid do?
3. What is different when you train logistic regression instead of the line?
4. The bank lowers the threshold from 0.5 to 0.3. What happens?
5. When is logistic regression not the right tool?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·