When the neighbour is slow: timeout, retry, circuit breaker

Code in:JavaC#PythonJavaScript

In short: A timeout stops waiting for a slow service, a retry with exponential backoff and jitter repeats only temporary errors, and a circuit breaker stops requests until the service recovers.

Problem

Thirty seconds that stopped the shop

On the morning of Black Friday the payment provider starts answering in 30 seconds instead of 300 ms. It does not go down: it answers, but slowly. A minute later the shop does not even show the catalogue, and the catalogue never calls the provider.

Here is why, in small. The shop has a pool of 4 threads (the lesson on thread pools). Every tick one payment and one catalogue request arrive. The provider answers in 30 ticks:

// a pool of 4 threads; every tick one payment and one catalogue request arrive
static void Run(string title, int timeout)
{
    var busyUntil = new int[4]; // until which tick each thread is busy
    int refused = 0;
    for (int t = 0; t < 8; t++)
    {
        int free = Array.FindIndex(busyUntil, u => u <= t);
        // the payment holds the thread until the provider answers
        if (free >= 0) busyUntil[free] = t + Math.Min(30, timeout);
        if (Array.FindIndex(busyUntil, u => u <= t) < 0) refused++; // the catalogue has no free thread
    }
    int busy = busyUntil.Count(u => u > 7);
    Console.WriteLine($"{title}: busy threads {busy} of 4, the catalogue refused {refused} requests");
}

Run("no timeout", int.MaxValue);

Output

no timeout: busy threads 4 of 4, the catalogue refused 5 requests

After four ticks all four threads wait for the provider. The catalogue refuses everything after that, although it has nothing to do with payments. A slow neighbour is more dangerous than a dead one: the dead one at least answers at once with an error.

The idea

Timeout: do not wait forever

A timeout is the longest time you wait for an answer. After it you stop, free the thread and tell the customer something clear: "The payment did not go through, the order is saved, try again in a moment." The same shop with a timeout of 2 ticks:

Run("timeout of 2 ticks", 2);

Output

timeout of 2 ticks: busy threads 2 of 4, the catalogue refused 0 requests

The payments still fail, because the provider is slow. But each one holds a thread for at most 2 ticks, there are always free threads, and the catalogue works.

How long should the timeout be? It comes from the latency budget: if the payment has 450 ms for p95, a 30 second timeout protects nothing. Usually it is a little above the neighbour's p99, measured, not guessed. And the timeout of the lower layer must be shorter than that of the upper one: otherwise the upper layer has given up while the lower one still works for nobody.

The idea

Retry: try again, but wisely

Many errors are temporary: a lost packet, a restarting server, a 503 answer ("I am busy"). A second attempt a little later often succeeds. This is a retry. Three rules make it safe:

1. Only temporary errors. Timeout and 503 are retried. "The card was declined" or 400 (a bad request) will be the same on the second attempt.
2. A wait that grows: 1, 2, 4, 8 ticks. This is exponential backoff: a neighbour who is drowning gets more and more air.
3. A limit on attempts. After it you give up and return the last error.

// only temporary errors: the provider may recover
static bool Retryable(string answer) =>
    answer is "timeout" or "503";

static string WithRetry(List<string> answers, int maxAttempts)
{
    int tick = 0;
    for (int attempt = 1; ; attempt++)
    {
        string answer = answers[attempt - 1]; // the recorded answer to attempt number attempt
        Console.WriteLine($"tick {tick}, attempt {attempt}: {answer}");
        if (!Retryable(answer) || attempt == maxAttempts) return answer;
        tick += 1 << (attempt - 1); // wait 1, 2, 4, 8… ticks
    }
}

The provider answers from a recorded script: timeout, then 503, then ok.

Console.WriteLine("result: " + WithRetry(["timeout", "503", "ok"], 4));

Output

tick 0, attempt 1: timeout
tick 1, attempt 2: 503
tick 3, attempt 3: ok
result: ok

Careful: a repeated payment can take the money twice if the first one went through and only the answer was lost. That is why the payment is retried only with an idempotency key: the provider recognises the repeated request and returns the first result.

Try it

Predict the output

The provider answers timeout three times and ok on the fourth attempt. The limit is 3 attempts. What does the program print?

Console.WriteLine("result: " + WithRetry(["timeout", "timeout", "timeout", "ok"], 3));

Answer

tick 0, attempt 1: timeout
tick 1, attempt 2: timeout
tick 2, attempt 3: timeout
result: timeout

tick 0, attempt 1: timeout
tick 1, attempt 2: timeout
tick 3, attempt 3: timeout
tick 7, attempt 4: ok
result: ok

tick 0, attempt 1: timeout
tick 1, attempt 2: timeout
tick 3, attempt 3: timeout
result: timeout

Check

Try it

Find the bug

A customer with a declined card waits a whole 7 ticks, and the bank sends four decline notifications. Which line has the bug?

Pick the line with the bug.

1// everything but ok is worth repeating?2static bool Retryable(string answer) =>3 answer != "ok";

Check

The idea

Jitter: so they do not all start together

When the provider fails, thousands of payments fail at once. With exponential backoff they all try again on tick 1, then on 3, then on 7: together, right when the provider is trying to recover. This is the thundering herd.

Jitter adds randomness to the wait: each client waits a random number of ticks, from 1 to double the wait. The program uses a small generator with a seed, so that it gives the same numbers in every language and on every run:

int seed = 7;

// a generator with a seed: the same numbers in every language
int Random(int bound)
{
    seed = (seed * 75 + 74) % 65537;
    return seed % bound;
}

string RetryTicks(bool jitter)
{
    var ticks = new List<int>();
    int tick = 0;
    for (int attempt = 0; attempt < 4; attempt++)
    {
        int wait = 1 << attempt; // 1, 2, 4, 8
        if (jitter) wait = 1 + Random(2 * wait); // from 1 to 2 · wait
        tick += wait;
        ticks.Add(tick);
    }
    return string.Join(" ", ticks);
}

foreach (var client in new[] { "A", "B", "C" })
    Console.WriteLine($"{client} no jitter: {RetryTicks(false)}");
foreach (var client in new[] { "A", "B", "C" })
    Console.WriteLine($"{client} jitter   : {RetryTicks(true)}");

Output

A no jitter: 1 3 7 15
B no jitter: 1 3 7 15
C no jitter: 1 3 7 15
A jitter   : 2 6 11 13
B jitter   : 2 5 12 19
C jitter   : 1 5 8 9

Without jitter the three hit the provider on the very same ticks. With jitter they spread out. In real code you do not set a seed: you use the language's generator, and in tests you pass your own so that the output is predictable.

The idea

Circuit breaker: stop calling

A retry helps with a short problem. If the provider has been down for ten minutes, every retry is one more request to someone who is drowning, and one more thread that waits. Then you need a circuit breaker, like the fuse in an electrical panel. It has three states:

• closed: requests go through. The breaker counts the failures in a row;
• open: after 3 failures in a row it stops calling. Every request is refused at once, without waiting;
• half-open: after 4 ticks it lets one probe through. If it succeeds, the breaker closes. If it fails, it opens again.

enum State { Closed, Open, HalfOpen }

class CircuitBreaker
{
    const int Threshold = 3; // failures in a row before it opens
    const int Cooldown = 4;  // ticks in OPEN before a probe
    public State State { get; private set; } = State.Closed;
    int failures;
    int openedAt;
    public int Calls { get; private set; } // how many times the provider was called

    public string Call(int tick, bool providerUp)
    {
        if (State == State.Open)
        {
            if (tick - openedAt < Cooldown)
                return "refused at once";
            State = State.HalfOpen; // one probe
        }
        Calls++;
        if (providerUp)
        {
            State = State.Closed;
            failures = 0;
            return "ok";
        }
        failures++;
        bool tooMany = failures >= Threshold;
        if (State == State.HalfOpen || tooMany)
        {
            State = State.Open;
            openedAt = tick;
        }
        return "error";
    }
}

See it

The breaker over 12 ticks

public string Call(int tick, bool providerUp)
{
    if (State == State.Open)
    {
        if (tick - openedAt < Cooldown)
            return "refused at once";
        State = State.HalfOpen; // one probe
    }
    Calls++;
    if (providerUp)
    {
        State = State.Closed;
        failures = 0;
        return "ok";
    }
    failures++;
    bool tooMany = failures >= Threshold;
    if (State == State.HalfOpen || tooMany)
    {
        State = State.Open;
        openedAt = tick;
    }
    return "error";
}

Step by step, in words

1. Closed: every call goes to the payment provider. The provider does not answer: tick 0 and tick 1 are errors, the counter is at 2 of 3.
2. Tick 2: the third error in a row. The breaker opens (Open) and remembers when.
3. Ticks 3, 4 and 5 are refused at once. They do not reach the provider and do not hold a thread. The shop can say "pay later" right away.
4. Tick 6: 4 ticks have passed. HalfOpen: one probe goes through. The provider is still down, so the breaker opens again.
5. From tick 7 the provider works again, but the breaker does not know yet. Ticks 7 to 9 are refused: the price of waiting.
6. Tick 10: the probe succeeds, the breaker closes (Closed) and the counter is reset. The provider was called 6 times out of 12.

The provider does not answer until tick 7 and works after that. The recorded script is tick >= 7:

var breaker = new CircuitBreaker();
for (int tick = 0; tick < 12; tick++)
{
    bool up = tick >= 7; // recorded: the provider is back on tick 7
    string result = breaker.Call(tick, up);
    Console.WriteLine($"tick {tick}: {result} → {breaker.State}");
}
Console.WriteLine($"calls to the provider: {breaker.Calls} of 12");

Output

tick 0: error → Closed
tick 1: error → Closed
tick 2: error → Open
tick 3: refused at once → Open
tick 4: refused at once → Open
tick 5: refused at once → Open
tick 6: error → Open
tick 7: refused at once → Open
tick 8: refused at once → Open
tick 9: refused at once → Open
tick 10: ok → Closed
tick 11: ok → Closed
calls to the provider: 6 of 12

Of 12 requests only 6 reach the provider. The others are refused at once: they hold no thread and put no load on someone who does not answer anyway. The price shows on ticks 7 to 9: the provider already works, and the breaker still refuses. A shorter cooldown brings the work back sooner, but hits the provider more often.

The property State is of type State. C# allows it: in State == State.Open the compiler understands that on the left is the property and on the right the enum.

Try it

Fill in

How many ticks does the open breaker wait before it lets a probe through?

if (State == State.Open)
{
    if (tick - openedAt < )
        return "refused at once";
    State = State.HalfOpen;
}

Check

The idea

Bulkhead and the answer on refusal

A bulkhead (as in the hull of a ship) is a separate thread pool for each neighbour: the payment has its own 4 threads and the catalogue its own. Then a slow provider eats only its own.

The breaker and the timeout say "not now", and the system must know what to do then. This is the fallback: the order is saved as "waiting for payment" and the customer pays later; the page shows the price from the cache; the payment goes through a second provider. Which answer is acceptable is for the business to decide, as with trade-offs.

The idea

When it hurts

• Retries on every layer. The gateway retries 3 times, the orders 3 times, the payment 3 times: up to 27 requests to the provider for one click. Retry in one place.
• Retrying an operation that is not idempotent. A double payment, two emails, two parcels. Without an idempotency key, do not retry.
• A timeout longer than the caller's. The upper layer has already told the customer "error", and the lower one still waits and retries.
• A breaker for the whole service because of one slow address. If only search is slow, do not stop the payment too.
• Numbers guessed, not measured. A threshold too low opens the breaker on every bit of noise; a cooldown too long keeps it open when the neighbour has long been working.

The same mechanisms are often provided by the infrastructure too (a service mesh such as Istio or Linkerd, an API gateway). It is taught in the DevOps course ZeroToOps. Here is the part in the code: what you do when the neighbour does not answer.

More precisely, if you are curious

The libraries. In a real project you do not write a breaker yourself: Java has Resilience4j (Spring Cloud Circuit Breaker uses it); C# has Polly and Microsoft.Extensions.Http.Resilience for HttpClient; Python has tenacity for retries and pybreaker; Node has cockatiel and opossum. Netflix's Hystrix is the historical model and is no longer developed.

The default timeouts. In .NET HttpClient waits 100 seconds by default. In Java's HttpClient and in requests for Python there is no timeout until you set one. In Node fetch (undici) waits for the headers up to 300 seconds. You met the network clients in the lesson "Network programming".

The deadline travels. If the client has 2 seconds in total and the gateway has already spent 0.5, the service below gets 1.5, not its own 2. gRPC does this with a deadline that travels with the request.

Retry-After and a retry budget. The answers 503 and 429 can say when to try (Retry-After). Many systems also limit retries overall, for example to 10% of the requests, so that they do not double the load exactly when it is worst.

Kinds of jitter. Here it is "full" jitter: random from 1 to double. There is also "equal" jitter (half fixed, half random) and "decorrelated" jitter. The goal is the same: the clients must not synchronise.

The generator in jitter is a local function in the top-level code and sees the variable seed outside it.

Exercise

Retries with a deadline

Solve the task in your own editor.

The customer waits for the payment at most 5 ticks. Write WithDeadline, which retries like WithRetry, but without a limit on attempts: it stops when the next attempt would come after the deadline, and says why. The provider answers timeout three times, then ok.

Output

tick 0, attempt 1: timeout
tick 1, attempt 2: timeout
tick 3, attempt 3: timeout
stopping: attempt 4 would be on tick 7, after the deadline 5
result: timeout

More tasks

• A breaker and retries together. Put the breaker in front of WithRetry. Is a refusal "at once" a temporary error?
• A bulkhead. Split the pool from the problem in two: 2 threads for payments and 2 for the catalogue. How many catalogue requests are refused now without a timeout?

Test

Check yourself

1. Why is a slow provider more dangerous than a provider that does not work at all?

The slow provider charges moreThe dead one answers at once with an error, while the slow one holds the threads until none are left for other requestsIt is not: both stop only the payment

2. Which errors are worth repeating?

None: retrying always hurtsThe temporary ones: timeout and 503. "The card was declined" will be the same on the second attemptAll that are not ok

3. Why is jitter added to the wait?

To hide in the logs how many times we triedTo make it fasterSo that thousands of clients do not retry on the same tick and hit the provider together, right when it is recovering

4. The breaker is open. What happens to a new request before the cooldown has passed?

The request goes to the provider, but the answer is thrown awayThe request waits until the breaker closesIt is refused at once, without calling the provider

5. Three layers on top of each other, each retries up to 3 times. How many requests can reach the bottom one if it does not answer?

3: as many as the attempts of one layer9: 3 + 3 + 327: 3 × 3 × 3

Check

Start the lesson over

Related lessons

• Builds on: Network programming · A pool of threads and tasks · API design: contract, versions, retried requests · Non-functional requirements you can measure
• Continues in: A transaction across services (saga) · Integrating old and foreign systems · Architecture review · Your first architecture task
• Tasks on this topic: The architect at work