Horizontal and vertical scaling in JavaScript

Problem

Black Friday and one server

On an ordinary day the shop gets about 50 requests a second and one server is enough. On Black Friday the requests are ten times more (an example, not a measurement). The CPU is at 100%, pages time out, and carts get lost. Last year the company bought a bigger machine. It cost twice as much, lasted a year, and it is still one: when it restarts for an update, the shop stops.

How do you add more power without waiting for a bigger machine, and without stopping the shop when one machine goes down?

The idea

A bigger machine or more machines

There are two ways:

Vertical scaling: a bigger machine, with more CPUs and memory. The code does not change. But the machine has a ceiling, the price grows faster than the power, and the server stays one: if it goes down, the shop stops.
Horizontal scaling: several identical copies of the server, and in front of them a balancer (load balancer) that shares the requests between them. There is no ceiling: you add a copy. One copy goes down and the others take over.

Horizontal scaling has a condition: every copy must be able to take every request. The balancer does not know what a request is about and hands them out in turn.

The balancer as real infrastructure (nginx, a cloud load balancer, Kubernetes) is a topic of the DevOps course ZeroToOps. Here it is an object that hands out the servers in turn (round robin), so that you can see what happens to the code.

See it

Three copies with memory

The server keeps the carts in its memory, the way the shop kept the sessions in a map in the lesson on sessions:

class AppServer {
  carts = new Map(); // the carts live in the memory of this server

  constructor(name) {
    this.name = name;
  }

  add(session, product) {
    if (!this.carts.has(session)) this.carts.set(session, []);
    const cart = this.carts.get(session);
    cart.push(product);
    return `${this.name}: ${cart.join(", ")}`;
  }
}

// the balancer hands out the servers in turn (round robin)
class Balancer {
  next = 0;

  constructor(servers) {
    this.servers = [...servers];
  }

  pick() {
    return this.servers[this.next++ % this.servers.length];
  }

  remove(server) {
    this.servers.splice(this.servers.indexOf(server), 1);
  }
}

Maria adds four products. The balancer gives the requests to A, B, C and A again:

const balancer = new Balancer([new AppServer("A"), new AppServer("B"), new AppServer("C")]);
console.log(balancer.pick().add("maria", "coffee"));
console.log(balancer.pick().add("maria", "tea"));
console.log(balancer.pick().add("maria", "chocolate"));
console.log(balancer.pick().add("maria", "honey"));

A: coffee
B: tea
C: chocolate
A: coffee, honey

Each copy sees its own cart. B has no coffee, C has no tea either, and A sees the coffee because Maria is with it again. At checkout the cart is whatever happens to be in the copy that takes the request.

The idea

A server that keeps nothing of its own

A server with state remembers something between two requests: the cart, the session, a counter. A server without state (stateless) remembers nothing of its own. Everything that must survive until the next request lives outside:

in a shared store for the sessions that all copies see (Redis);
in the database;
with the client, signed so that it cannot be changed (in the lesson on JWT).

Then the copies can stand in for each other: each takes every request, and when one stops, nothing is lost.

See it

The cart in the shared store

// the shared session store (Redis in a real shop)
class SessionStore {
  carts = new Map();

  add(session, product) {
    if (!this.carts.has(session)) this.carts.set(session, []);
    const cart = this.carts.get(session);
    cart.push(product);
    return [...cart];
  }
}

class AppServer {
  constructor(name, store) {
    this.name = name;
    this.store = store; // the server keeps nothing of its own
  }

  add(session, product) {
    const cart = this.store.add(session, product);
    return `${this.name}: ${cart.join(", ")}`;
  }
}

The same requests, and after the third one server B goes down:

const store = new SessionStore();
const b = new AppServer("B", store);
const balancer = new Balancer([new AppServer("A", store), b, new AppServer("C", store)]);
console.log(balancer.pick().add("maria", "coffee"));
console.log(balancer.pick().add("maria", "tea"));
console.log(balancer.pick().add("maria", "chocolate"));
balancer.remove(b); // server B goes down
console.log(balancer.pick().add("maria", "honey"));

A: coffee
B: coffee, tea
C: coffee, tea, chocolate
C: coffee, tea, chocolate, honey

Every copy sees the whole cart. When B goes down, the balancer gives the request to C, and the cart is whole, because it was never in B. list(cart) and [...cart] return a copy: the server cannot touch the list in the store, just as it could not with a real Redis.

Try it

Predict the output

Two servers with memory, Ivan adds three products. What does the last line print?

const balancer = new Balancer([new AppServer("A"), new AppServer("B")]);
balancer.pick().add("ivan", "coffee");
balancer.pick().add("ivan", "tea");
console.log(balancer.pick().add("ivan", "honey"));

Answer

B: tea, honey

A: coffee, tea, honey

A: coffee, honey

The idea

The database does not multiply so easily

The copies of the server write to one database. Multiplying the database is harder, because the writes must be in one place, or two writes fight.

The usual step is a read replica: a copy of the database that gets the changes of the primary and answers only reads. Most requests in a shop are reads (the catalogue, pages, history), so the replicas take a large part of the load. How the changes are sent, you saw in the lesson on replication.

The price is the replication lag: a change reaches the replica a little later. Usually fractions of a second, but under load seconds too. Here the lag is 3 ticks:

// the primary database: every write goes through it
class Primary {
  log = [];

  insert(order, customer, now) {
    this.log.push({ at: now, order, customer });
  }

  orders(customer) {
    return this.log.filter((c) => c.customer === customer).map((c) => c.order);
  }
}

// a read replica: a change reaches it lag ticks after the write
class Replica {
  constructor(primary, lag) {
    this.primary = primary;
    this.lag = lag;
  }

  orders(customer, now) {
    const ids = [];
    for (const c of this.primary.log) {
      const arrived = c.at + this.lag <= now;
      if (arrived && c.customer === customer)
        ids.push(c.order);
    }
    return ids;
  }
}

function show(ids) {
  return ids.length === 0 ? "no orders" : `orders [${ids.join(", ")}]`;
}

Maria pays on tick 0 and at once opens "My orders", which reads from the replica:

const primary = new Primary();
const replica = new Replica(primary, 3);
primary.insert(1042, "maria", 0); // Maria pays on tick 0
for (let now = 0; now <= 3; now++) {
  console.log(`tick ${now}, replica: ${show(replica.orders("maria", now))}`);
}

tick 0, replica: no orders
tick 1, replica: no orders
tick 2, replica: no orders
tick 3, replica: orders [1042]

For three ticks she does not see her order. She pays a second time, or writes to support that the money is gone and there is no order.

See it

Three copies, a shared store and the replica

myOrders(customer, now) {
  if (this.justWrote(customer, now))
    return this.primary.orders(customer);
  return this.replica.orders(customer, now);
}

justWrote(customer, now) {
  const last = this.lastWrite.get(customer);
  return last !== undefined && now - last < this.lag;
}

orders(customer, now) {
  const ids = [];
  for (const c of this.primary.log) {
    const arrived = c.at + this.lag <= now;
    if (arrived && c.customer === customer)
      ids.push(c.order);
  }
  return ids;
}

myOrders(customer, now) {
  if (this.justWrote(customer, now))
    return this.primary.orders(customer);
  return this.replica.orders(customer, now);
}

justWrote(customer, now) {
  const last = this.lastWrite.get(customer);
  return last !== undefined && now - last < this.lag;
}

Step by step, in words

Maria adds coffee. The balancer sends the request to A, and the cart stays in the memory of A.
The next request (tea) goes to B. B knows nothing about the coffee: its cart holds only tea.
The fix: the servers keep nothing. The cart lives in a shared store, and every server reads it from there.
Now C adds chocolate and sees the coffee and the tea. It does not matter which server takes the request.
B goes down. The balancer drops it, and the cart survives: it was never in B.
Maria pays: order 1042 is written to the primary on tick 0. The replica gets the change 3 ticks later.
On tick 1 she opens "My orders". The read goes to the replica, and the order is not there yet.
On tick 3 the change reaches the replica and the order shows up. Until then Maria thinks the payment failed.
The fix: whoever has just written reads from the primary for 3 ticks. Everyone else reads from the replica.
Servers without state multiply. The database for writes stays one, and its replicas are always a little behind.

The idea

Whoever wrote reads from the primary

Most people do not notice that the catalogue is half a second old. But people expect to see their own change at once. So: whoever has just written reads from the primary until the lag has passed. Everyone else reads from the replica. This is called read-your-writes.

// whoever has just written reads from the primary
class OrderService {
  lastWrite = new Map();

  constructor(primary, replica, lag) {
    this.primary = primary;
    this.replica = replica;
    this.lag = lag;
  }

  place(order, customer, now) {
    this.primary.insert(order, customer, now);
    this.lastWrite.set(customer, now);
  }

  myOrders(customer, now) {
    if (this.justWrote(customer, now))
      return this.primary.orders(customer);
    return this.replica.orders(customer, now);
  }

  justWrote(customer, now) {
    const last = this.lastWrite.get(customer);
    return last !== undefined && now - last < this.lag;
  }
}

function read(orders, customer, now) {
  const from = orders.justWrote(customer, now) ? "primary" : "replica";
  return `tick ${now}, ${customer} → ${from}: ${show(orders.myOrders(customer, now))}`;
}

const primary = new Primary();
const replica = new Replica(primary, 3);
const orders = new OrderService(primary, replica, 3);
orders.place(1042, "maria", 0);
for (let now = 0; now <= 3; now++) {
  console.log(read(orders, "maria", now));
}
console.log(read(orders, "ivan", 1));

tick 0, maria → primary: orders [1042]
tick 1, maria → primary: orders [1042]
tick 2, maria → primary: orders [1042]
tick 3, maria → replica: orders [1042]
tick 1, ivan → replica: no orders

Maria sees her order at once, from the primary. Ivan wrote nothing and reads from the replica: the load on the primary grows only with the people who have just written.

The idea

What scales and when it hurts

Scales: the application servers, when they have no state; the reads, with replicas and a cache (in the lesson on caching).
Does not scale easily: the writes to the primary database. When one database is not enough for the writes, the data is split into parts (sharding) or split into services with their own databases (in the lesson on the monolith and services). Both are big decisions.
Hidden state: a cache in the memory of each copy (each has its own old price), files uploaded to the disk of one copy, counters in memory, a scheduled job that starts on every copy and sends three emails instead of one.
When it hurts: copies need a shared store, replicas and monitoring of every copy. If one server is enough, horizontal scaling is complexity without a gain. First you measure where the bottleneck is. Sometimes the bigger machine is the cheaper first move, and often an index in the database is.

Try it

Find the bug

The shop allows only 5 login attempts and then blocks. The penetration test shows that with three copies the attacker tries 15 passwords. Which line has the bug?

Pick the line with the bug.

More precisely, if you are curious

Sticky sessions. The balancer can send the same buyer to the same copy every time (by a cookie). That patches the cart, but the copies still cannot stand in for each other: if a copy goes down, its carts are gone, and the load is shared unevenly.

Other ways to read your writes. The client remembers the number of its last write and reads from a replica that already has it. Or all reads of "my" data (profile, orders) always go to the primary. Or replication is synchronous for the important writes: slower, but without lag.

Autoscaling. The cloud adds and removes copies by the load. That, the balancer, the checks whether a copy is alive and Kubernetes are infrastructure, so they are in the DevOps course ZeroToOps.

The real things: sessions in Redis are ready-made in Spring Session, in ASP.NET Core with AddStackExchangeRedisCache and AddSession, in Django with SESSION_ENGINE, in Express with connect-redis. PostgreSQL, MySQL and SQL Server have read replicas, and frameworks let you route the reads to them.

Exercise

The shared attempt counter

Solve it in your editor.

Fix the bug above: the attempt counter is one class Attempts, shared by all copies. The server asks it and returns "blocked" after the fifth attempt. Run three copies behind the balancer and seven wrong logins as maria.

attempt 1 (A): wrong password
attempt 2 (B): wrong password
attempt 3 (C): wrong password
attempt 4 (A): wrong password
attempt 5 (B): wrong password
attempt 6 (C): blocked
attempt 7 (A): blocked

The sixth and seventh attempts are blocked, although C and A see them for the second time.

More tasks

Expiry. Add a tick clock and reset the counter 10 ticks after the first attempt.
Uploaded files. The server keeps the product photo on its own disk. What does a buyer see when the balancer sends them to another copy? Where should the photo live?

Test

Check yourself

1. Which one is horizontal scaling?More copies of the same server, with a balancer sharing the requests between themA faster programming languageA bigger machine with more CPUs and memory

2. Why does the cart disappear when there are three copies of the server?The balancer deletes the cookiesThe three copies write to the same memory at onceThe cart is in the memory of one copy, and the next request goes to another

3. Maria pays and at once opens "My orders", and the order is not there. What most likely happened?The payment failedThe browser cache keeps the old pageThe write is in the primary, and the read is from a replica the change has not reached yet

4. The limit is 5 login attempts, and the counter is in the memory of each server. How many attempts does an attacker get with three copies?5UnlimitedUp to 15

5. What does NOT get faster when you add copies of the server?Drawing the pagesThe writes to the primary databaseHandling requests that read from the cache

Scaling

Cookies