Skip to content
wannacode

Scaling

In short: Horizontal scaling adds copies of the server behind a load balancer; it works when the server keeps no state of its own, while database replicas lag behind.

Problem

Black Friday and one server

On an ordinary day the shop gets about 50 requests a second and one server is enough. On Black Friday the requests are ten times more (an example, not a measurement). The CPU is at 100%, pages time out, and carts get lost. Last year the company bought a bigger machine. It cost twice as much, lasted a year, and it is still one: when it restarts for an update, the shop stops.

How do you add more power without waiting for a bigger machine, and without stopping the shop when one machine goes down?

The idea

A bigger machine or more machines

There are two ways:

  • Vertical scaling: a bigger machine, with more CPUs and memory. The code does not change. But the machine has a ceiling, the price grows faster than the power, and the server stays one: if it goes down, the shop stops.
  • Horizontal scaling: several identical copies of the server, and in front of them a balancer (load balancer) that shares the requests between them. There is no ceiling: you add a copy. One copy goes down and the others take over.

Horizontal scaling has a condition: every copy must be able to take every request. The balancer does not know what a request is about and hands them out in turn.

The balancer as real infrastructure (nginx, a cloud load balancer, Kubernetes) is a topic of the DevOps course ZeroToOps. Here it is an object that hands out the servers in turn (round robin), so that you can see what happens to the code.

See it

Three copies with memory

The server keeps the carts in its memory, the way the shop kept the sessions in a map in the lesson on sessions:

class AppServer(string name)
{
    // the carts live in the memory of this server
    private readonly Dictionary<string, List<string>> carts = new();


    public string Add(string session, string product)
    {
        if (!carts.TryGetValue(session, out var cart))
            carts[session] = cart = [];
        cart.Add(product);
        return $"{name}: {string.Join(", ", cart)}";
    }
}
// the balancer hands out the servers in turn (round robin)
class Balancer(List<AppServer> servers)
{
    private int next;


    public AppServer Pick() => servers[next++ % servers.Count];


    public void Remove(AppServer server) => servers.Remove(server);
}

Maria adds four products. The balancer gives the requests to A, B, C and A again:

var balancer = new Balancer([new AppServer("A"), new AppServer("B"), new AppServer("C")]);
Console.WriteLine(balancer.Pick().Add("maria", "coffee"));
Console.WriteLine(balancer.Pick().Add("maria", "tea"));
Console.WriteLine(balancer.Pick().Add("maria", "chocolate"));
Console.WriteLine(balancer.Pick().Add("maria", "honey"));
Output
A: coffee
B: tea
C: chocolate
A: coffee, honey

Each copy sees its own cart. B has no coffee, C has no tea either, and A sees the coffee because Maria is with it again. At checkout the cart is whatever happens to be in the copy that takes the request.

The idea

A server that keeps nothing of its own

A server with state remembers something between two requests: the cart, the session, a counter. A server without state (stateless) remembers nothing of its own. Everything that must survive until the next request lives outside:

  • in a shared store for the sessions that all copies see (Redis);
  • in the database;
  • with the client, signed so that it cannot be changed (in the lesson on JWT).

Then the copies can stand in for each other: each takes every request, and when one stops, nothing is lost.

See it

The cart in the shared store

// the shared session store (Redis in a real shop)
class SessionStore
{
    private readonly Dictionary<string, List<string>> carts = new();


    public List<string> Add(string session, string product)
    {
        if (!carts.TryGetValue(session, out var cart))
            carts[session] = cart = [];
        cart.Add(product);
        return [.. cart];
    }
}
// the server keeps nothing of its own
class AppServer(string name, SessionStore store)
{
    public string Add(string session, string product)
    {
        var cart = store.Add(session, product);
        return $"{name}: {string.Join(", ", cart)}";
    }
}

The same requests, and after the third one server B goes down:

var store = new SessionStore();
var b = new AppServer("B", store);
var balancer = new Balancer([new AppServer("A", store), b, new AppServer("C", store)]);
Console.WriteLine(balancer.Pick().Add("maria", "coffee"));
Console.WriteLine(balancer.Pick().Add("maria", "tea"));
Console.WriteLine(balancer.Pick().Add("maria", "chocolate"));
balancer.Remove(b); // server B goes down
Console.WriteLine(balancer.Pick().Add("maria", "honey"));
Output
A: coffee
B: coffee, tea
C: coffee, tea, chocolate
C: coffee, tea, chocolate, honey

Every copy sees the whole cart. When B goes down, the balancer gives the request to C, and the cart is whole, because it was never in B. List.copyOf and [.. cart] return a copy: the server cannot touch the list in the store, just as it could not with a real Redis.

Try it

Predict the output

Two servers with memory, Ivan adds three products. What does the last line print?

var balancer = new Balancer([new AppServer("A"), new AppServer("B")]);
balancer.Pick().Add("ivan", "coffee");
balancer.Pick().Add("ivan", "tea");
Console.WriteLine(balancer.Pick().Add("ivan", "honey"));
Answer

The idea

The database does not multiply so easily

The copies of the server write to one database. Multiplying the database is harder, because the writes must be in one place, or two writes fight.

The usual step is a read replica: a copy of the database that gets the changes of the primary and answers only reads. Most requests in a shop are reads (the catalogue, pages, history), so the replicas take a large part of the load. How the changes are sent, you saw in the lesson on replication.

The price is the replication lag: a change reaches the replica a little later. Usually fractions of a second, but under load seconds too. Here the lag is 3 ticks:

record Change(int At, int Order, string Customer);


// the primary database: every write goes through it
class Primary
{
    public List<Change> Log { get; } = [];


    public void Insert(int order, string customer, int now) =>
        Log.Add(new Change(now, order, customer));


    public List<int> Orders(string customer) =>
        Log.Where(c => c.Customer == customer).Select(c => c.Order).ToList();
}


// a read replica: a change reaches it lag ticks after the write
class Replica(Primary primary, int lag)
{
    public List<int> Orders(string customer, int now)
    {
        var ids = new List<int>();
        foreach (var c in primary.Log)
        {
            var arrived = c.At + lag <= now;
            if (arrived && c.Customer == customer)
                ids.Add(c.Order);
        }
        return ids;
    }
}
static string Show(List<int> ids) =>
    ids.Count == 0 ? "no orders" : $"orders [{string.Join(", ", ids)}]";

Maria pays on tick 0 and at once opens "My orders", which reads from the replica:

var primary = new Primary();
var replica = new Replica(primary, 3);
primary.Insert(1042, "maria", 0); // Maria pays on tick 0
for (var now = 0; now <= 3; now++)
{
    Console.WriteLine($"tick {now}, replica: {Show(replica.Orders("maria", now))}");
}
Output
tick 0, replica: no orders
tick 1, replica: no orders
tick 2, replica: no orders
tick 3, replica: orders [1042]

For three ticks she does not see her order. She pays a second time, or writes to support that the money is gone and there is no order.

See it

Three copies, a shared store and the replica

public List<int> MyOrders(string customer, int now)
{
    if (JustWrote(customer, now))
        return primary.Orders(customer);
    return replica.Orders(customer, now);
}


public bool JustWrote(string customer, int now) =>
    lastWrite.TryGetValue(customer, out var last)
        && now - last < lag;
Step by step, in words
  1. Maria adds coffee. The balancer sends the request to A, and the cart stays in the memory of A.
  2. The next request (tea) goes to B. B knows nothing about the coffee: its cart holds only tea.
  3. The fix: the servers keep nothing. The cart lives in a shared store, and every server reads it from there.
  4. Now C adds chocolate and sees the coffee and the tea. It does not matter which server takes the request.
  5. B goes down. The balancer drops it, and the cart survives: it was never in B.
  6. Maria pays: order 1042 is written to the primary on tick 0. The replica gets the change 3 ticks later.
  7. On tick 1 she opens "My orders". The read goes to the replica, and the order is not there yet.
  8. On tick 3 the change reaches the replica and the order shows up. Until then Maria thinks the payment failed.
  9. The fix: whoever has just written reads from the primary for 3 ticks. Everyone else reads from the replica.
  10. Servers without state multiply. The database for writes stays one, and its replicas are always a little behind.

The idea

Whoever wrote reads from the primary

Most people do not notice that the catalogue is half a second old. But people expect to see their own change at once. So: whoever has just written reads from the primary until the lag has passed. Everyone else reads from the replica. This is called read-your-writes.

// whoever has just written reads from the primary
class OrderService(Primary primary, Replica replica, int lag)
{
    private readonly Dictionary<string, int> lastWrite = new();


    public void Place(int order, string customer, int now)
    {
        primary.Insert(order, customer, now);
        lastWrite[customer] = now;
    }


    public List<int> MyOrders(string customer, int now)
    {
        if (JustWrote(customer, now))
            return primary.Orders(customer);
        return replica.Orders(customer, now);
    }


    public bool JustWrote(string customer, int now) =>
        lastWrite.TryGetValue(customer, out var last)
            && now - last < lag;
}
static string Read(OrderService orders, string customer, int now)
{
    var from = orders.JustWrote(customer, now) ? "primary" : "replica";
    return $"tick {now}, {customer} → {from}: {Show(orders.MyOrders(customer, now))}";
}
var primary = new Primary();
var replica = new Replica(primary, 3);
var orders = new OrderService(primary, replica, 3);
orders.Place(1042, "maria", 0);
for (var now = 0; now <= 3; now++)
{
    Console.WriteLine(Read(orders, "maria", now));
}
Console.WriteLine(Read(orders, "ivan", 1));
Output
tick 0, maria → primary: orders [1042]
tick 1, maria → primary: orders [1042]
tick 2, maria → primary: orders [1042]
tick 3, maria → replica: orders [1042]
tick 1, ivan → replica: no orders

Maria sees her order at once, from the primary. Ivan wrote nothing and reads from the replica: the load on the primary grows only with the people who have just written.

The idea

What scales and when it hurts

  • Scales: the application servers, when they have no state; the reads, with replicas and a cache (in the lesson on caching).
  • Does not scale easily: the writes to the primary database. When one database is not enough for the writes, the data is split into parts (sharding) or split into services with their own databases (in the lesson on the monolith and services). Both are big decisions.
  • Hidden state: a cache in the memory of each copy (each has its own old price), files uploaded to the disk of one copy, counters in memory, a scheduled job that starts on every copy and sends three emails instead of one.
  • When it hurts: copies need a shared store, replicas and monitoring of every copy. If one server is enough, horizontal scaling is complexity without a gain. First you measure where the bottleneck is. Sometimes the bigger machine is the cheaper first move, and often an index in the database is.

Try it

Find the bug

The shop allows only 5 login attempts and then blocks. The penetration test shows that with three copies the attacker tries 15 passwords. Which line has the bug?

Pick the line with the bug.

More precisely, if you are curious

Sticky sessions. The balancer can send the same buyer to the same copy every time (by a cookie). That patches the cart, but the copies still cannot stand in for each other: if a copy goes down, its carts are gone, and the load is shared unevenly.

Other ways to read your writes. The client remembers the number of its last write and reads from a replica that already has it. Or all reads of "my" data (profile, orders) always go to the primary. Or replication is synchronous for the important writes: slower, but without lag.

Autoscaling. The cloud adds and removes copies by the load. That, the balancer, the checks whether a copy is alive and Kubernetes are infrastructure, so they are in the DevOps course ZeroToOps.

The real things: sessions in Redis are ready-made in Spring Session, in ASP.NET Core with AddStackExchangeRedisCache and AddSession, in Django with SESSION_ENGINE, in Express with connect-redis. PostgreSQL, MySQL and SQL Server have read replicas, and frameworks let you route the reads to them.

Exercise

The shared attempt counter

Solve it in your editor.

Fix the bug above: the attempt counter is one class Attempts, shared by all copies. The server asks it and returns "blocked" after the fifth attempt. Run three copies behind the balancer and seven wrong logins as maria.

Output
attempt 1 (A): wrong password
attempt 2 (B): wrong password
attempt 3 (C): wrong password
attempt 4 (A): wrong password
attempt 5 (B): wrong password
attempt 6 (C): blocked
attempt 7 (A): blocked

The sixth and seventh attempts are blocked, although C and A see them for the second time.

More tasks

  • Expiry. Add a tick clock and reset the counter 10 ticks after the first attempt.
  • Uploaded files. The server keeps the product photo on its own disk. What does a buyer see when the balancer sends them to another copy? Where should the photo live?

Test

Check yourself

1. Which one is horizontal scaling?
2. Why does the cart disappear when there are three copies of the server?
3. Maria pays and at once opens "My orders", and the order is not there. What most likely happened?
4. The limit is 5 login attempts, and the counter is in the memory of each server. How many attempts does an attacker get with three copies?
5. What does NOT get faster when you add copies of the server?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·