Skip to content
wannacode

When the neighbour is slow: timeout, retry, circuit breaker

In short: A timeout stops waiting for a slow service, a retry with exponential backoff and jitter repeats only temporary errors, and a circuit breaker stops requests until the service recovers.

Problem

Thirty seconds that stopped the shop

On the morning of Black Friday the payment provider starts answering in 30 seconds instead of 300 ms. It does not go down: it answers, but slowly. A minute later the shop does not even show the catalogue, and the catalogue never calls the provider.

Here is why, in small. The shop has a pool of 4 threads (the lesson on thread pools). Every tick one payment and one catalogue request arrive. The provider answers in 30 ticks:

# a pool of 4 threads; every tick one payment and one catalogue request arrive
def run(title, timeout):
    busy_until = [0] * 4  # until which tick each thread is busy
    refused = 0
    for t in range(8):
        free = free_thread(busy_until, t)
        # the payment holds the thread until the provider answers
        if free is not None:
            busy_until[free] = t + min(30, timeout)
        if free_thread(busy_until, t) is None:
            refused += 1  # the catalogue has no free thread
    busy = sum(1 for u in busy_until if u > 7)
    print(f"{title}: busy threads {busy} of 4, the catalogue refused {refused} requests")




def free_thread(busy_until, t):
    return next((i for i, u in enumerate(busy_until) if u <= t), None)
run("no timeout", math.inf)
Output
no timeout: busy threads 4 of 4, the catalogue refused 5 requests

After four ticks all four threads wait for the provider. The catalogue refuses everything after that, although it has nothing to do with payments. A slow neighbour is more dangerous than a dead one: the dead one at least answers at once with an error.

The idea

Timeout: do not wait forever

A timeout is the longest time you wait for an answer. After it you stop, free the thread and tell the customer something clear: "The payment did not go through, the order is saved, try again in a moment." The same shop with a timeout of 2 ticks:

run("timeout of 2 ticks", 2)
Output
timeout of 2 ticks: busy threads 2 of 4, the catalogue refused 0 requests

The payments still fail, because the provider is slow. But each one holds a thread for at most 2 ticks, there are always free threads, and the catalogue works.

How long should the timeout be? It comes from the latency budget: if the payment has 450 ms for p95, a 30 second timeout protects nothing. Usually it is a little above the neighbour's p99, measured, not guessed. And the timeout of the lower layer must be shorter than that of the upper one: otherwise the upper layer has given up while the lower one still works for nobody.

The idea

Retry: try again, but wisely

Many errors are temporary: a lost packet, a restarting server, a 503 answer ("I am busy"). A second attempt a little later often succeeds. This is a retry. Three rules make it safe:

  1. Only temporary errors. Timeout and 503 are retried. "The card was declined" or 400 (a bad request) will be the same on the second attempt.
  2. A wait that grows: 1, 2, 4, 8 ticks. This is exponential backoff: a neighbour who is drowning gets more and more air.
  3. A limit on attempts. After it you give up and return the last error.
# only temporary errors: the provider may recover
def retryable(answer):
    return answer in ("timeout", "503")
def with_retry(answers, max_attempts):
    tick = 0
    for attempt in range(1, max_attempts + 1):
        answer = answers[attempt - 1]  # the recorded answer to attempt number attempt
        print(f"tick {tick}, attempt {attempt}: {answer}")
        if not retryable(answer) or attempt == max_attempts:
            return answer
        tick += 2 ** (attempt - 1)  # wait 1, 2, 4, 8… ticks

The provider answers from a recorded script: timeout, then 503, then ok.

print("result:", with_retry(["timeout", "503", "ok"], 4))
Output
tick 0, attempt 1: timeout
tick 1, attempt 2: 503
tick 3, attempt 3: ok
result: ok

Careful: a repeated payment can take the money twice if the first one went through and only the answer was lost. That is why the payment is retried only with an idempotency key: the provider recognises the repeated request and returns the first result.

Try it

Predict the output

The provider answers timeout three times and ok on the fourth attempt. The limit is 3 attempts. What does the program print?

print("result:", with_retry(["timeout", "timeout", "timeout", "ok"], 3))
Answer

Try it

Find the bug

A customer with a declined card waits a whole 7 ticks, and the bank sends four decline notifications. Which line has the bug?

Pick the line with the bug.

The idea

Jitter: so they do not all start together

When the provider fails, thousands of payments fail at once. With exponential backoff they all try again on tick 1, then on 3, then on 7: together, right when the provider is trying to recover. This is the thundering herd.

Jitter adds randomness to the wait: each client waits a random number of ticks, from 1 to double the wait. The program uses a small generator with a seed, so that it gives the same numbers in every language and on every run:

seed = 7




# a generator with a seed: the same numbers in every language
def random(bound):
    global seed
    seed = (seed * 75 + 74) % 65537
    return seed % bound




def retry_ticks(jitter):
    ticks = []
    tick = 0
    for attempt in range(4):
        wait = 2 ** attempt  # 1, 2, 4, 8
        if jitter:
            wait = 1 + random(2 * wait)  # from 1 to 2 · wait
        tick += wait
        ticks.append(str(tick))
    return " ".join(ticks)
for client in ["A", "B", "C"]:
    print(f"{client} no jitter: {retry_ticks(False)}")
for client in ["A", "B", "C"]:
    print(f"{client} jitter   : {retry_ticks(True)}")
Output
A no jitter: 1 3 7 15
B no jitter: 1 3 7 15
C no jitter: 1 3 7 15
A jitter   : 2 6 11 13
B jitter   : 2 5 12 19
C jitter   : 1 5 8 9

Without jitter the three hit the provider on the very same ticks. With jitter they spread out. In real code you do not set a seed: you use the language's generator, and in tests you pass your own so that the output is predictable.

The idea

Circuit breaker: stop calling

A retry helps with a short problem. If the provider has been down for ten minutes, every retry is one more request to someone who is drowning, and one more thread that waits. Then you need a circuit breaker, like the fuse in an electrical panel. It has three states:

  • closed: requests go through. The breaker counts the failures in a row;
  • open: after 3 failures in a row it stops calling. Every request is refused at once, without waiting;
  • half-open: after 4 ticks it lets one probe through. If it succeeds, the breaker closes. If it fails, it opens again.
THRESHOLD = 3  # failures in a row before it opens
COOLDOWN = 4  # ticks in OPEN before a probe




class State(Enum):
    CLOSED = 1
    OPEN = 2
    HALF_OPEN = 3




class CircuitBreaker:
    def __init__(self):
        self.state = State.CLOSED
        self.failures = 0
        self.opened_at = 0
        self.calls = 0  # how many times the provider was called


    def call(self, tick, provider_up):
        if self.state == State.OPEN:
            if tick - self.opened_at < COOLDOWN:
                return "refused at once"
            # one probe
            self.state = State.HALF_OPEN
        self.calls += 1
        if provider_up:
            self.state = State.CLOSED
            self.failures = 0
            return "ok"
        self.failures += 1
        too_many = self.failures >= THRESHOLD
        if self.state == State.HALF_OPEN or too_many:
            self.state = State.OPEN
            self.opened_at = tick
        return "error"

See it

The breaker over 12 ticks

def call(self, tick, provider_up):
    if self.state == State.OPEN:
        if tick - self.opened_at < COOLDOWN:
            return "refused at once"
        # one probe
        self.state = State.HALF_OPEN
    self.calls += 1
    if provider_up:
        self.state = State.CLOSED
        self.failures = 0
        return "ok"
    self.failures += 1
    too_many = self.failures >= THRESHOLD
    if self.state == State.HALF_OPEN or too_many:
        self.state = State.OPEN
        self.opened_at = tick
    return "error"
Step by step, in words
  1. CLOSED: every call goes to the payment provider. The provider does not answer: tick 0 and tick 1 are errors, the counter is at 2 of 3.
  2. Tick 2: the third error in a row. The breaker opens (OPEN) and remembers when.
  3. Ticks 3, 4 and 5 are refused at once. They do not reach the provider and do not hold a thread. The shop can say "pay later" right away.
  4. Tick 6: 4 ticks have passed. HALF_OPEN: one probe goes through. The provider is still down, so the breaker opens again.
  5. From tick 7 the provider works again, but the breaker does not know yet. Ticks 7 to 9 are refused: the price of waiting.
  6. Tick 10: the probe succeeds, the breaker closes (CLOSED) and the counter is reset. The provider was called 6 times out of 12.

The provider does not answer until tick 7 and works after that. The recorded script is tick >= 7:

breaker = CircuitBreaker()
for tick in range(12):
    up = tick >= 7  # recorded: the provider is back on tick 7
    result = breaker.call(tick, up)
    print(f"tick {tick}: {result} → {breaker.state.name}")
print(f"calls to the provider: {breaker.calls} of 12")
Output
tick 0: error → CLOSED
tick 1: error → CLOSED
tick 2: error → OPEN
tick 3: refused at once → OPEN
tick 4: refused at once → OPEN
tick 5: refused at once → OPEN
tick 6: error → OPEN
tick 7: refused at once → OPEN
tick 8: refused at once → OPEN
tick 9: refused at once → OPEN
tick 10: ok → CLOSED
tick 11: ok → CLOSED
calls to the provider: 6 of 12

Of 12 requests only 6 reach the provider. The others are refused at once: they hold no thread and put no load on someone who does not answer anyway. The price shows on ticks 7 to 9: the provider already works, and the breaker still refuses. A shorter cooldown brings the work back sooner, but hits the provider more often.

State(Enum) gives the states names, and .name prints them: CLOSED, OPEN, HALF_OPEN.

Try it

Fill in

How many ticks does the open breaker wait before it lets a probe through?

if self.state == State.OPEN:
    if tick - self.opened_at < :
        return "refused at once"
    self.state = State.HALF_OPEN

The idea

Bulkhead and the answer on refusal

A bulkhead (as in the hull of a ship) is a separate thread pool for each neighbour: the payment has its own 4 threads and the catalogue its own. Then a slow provider eats only its own.

The breaker and the timeout say "not now", and the system must know what to do then. This is the fallback: the order is saved as "waiting for payment" and the customer pays later; the page shows the price from the cache; the payment goes through a second provider. Which answer is acceptable is for the business to decide, as with trade-offs.

The idea

When it hurts

  • Retries on every layer. The gateway retries 3 times, the orders 3 times, the payment 3 times: up to 27 requests to the provider for one click. Retry in one place.
  • Retrying an operation that is not idempotent. A double payment, two emails, two parcels. Without an idempotency key, do not retry.
  • A timeout longer than the caller's. The upper layer has already told the customer "error", and the lower one still waits and retries.
  • A breaker for the whole service because of one slow address. If only search is slow, do not stop the payment too.
  • Numbers guessed, not measured. A threshold too low opens the breaker on every bit of noise; a cooldown too long keeps it open when the neighbour has long been working.

The same mechanisms are often provided by the infrastructure too (a service mesh such as Istio or Linkerd, an API gateway). It is taught in the DevOps course ZeroToOps. Here is the part in the code: what you do when the neighbour does not answer.

More precisely, if you are curious

The libraries. In a real project you do not write a breaker yourself: Java has Resilience4j (Spring Cloud Circuit Breaker uses it); C# has Polly and Microsoft.Extensions.Http.Resilience for HttpClient; Python has tenacity for retries and pybreaker; Node has cockatiel and opossum. Netflix's Hystrix is the historical model and is no longer developed.

The default timeouts. In .NET HttpClient waits 100 seconds by default. In Java's HttpClient and in requests for Python there is no timeout until you set one. In Node fetch (undici) waits for the headers up to 300 seconds. You met the network clients in the lesson "Network programming".

The deadline travels. If the client has 2 seconds in total and the gateway has already spent 0.5, the service below gets 1.5, not its own 2. gRPC does this with a deadline that travels with the request.

Retry-After and a retry budget. The answers 503 and 429 can say when to try (Retry-After). Many systems also limit retries overall, for example to 10% of the requests, so that they do not double the load exactly when it is worst.

Kinds of jitter. Here it is "full" jitter: random from 1 to double. There is also "equal" jitter (half fixed, half random) and "decorrelated" jitter. The goal is the same: the clients must not synchronise.

global seed lets the function change the module's variable. In a class, or in an object from random.Random(7), the state is tidier.

Exercise

Retries with a deadline

Solve the task in your own editor.

The customer waits for the payment at most 5 ticks. Write with_deadline, which retries like with_retry, but without a limit on attempts: it stops when the next attempt would come after the deadline, and says why. The provider answers timeout three times, then ok.

Output
tick 0, attempt 1: timeout
tick 1, attempt 2: timeout
tick 3, attempt 3: timeout
stopping: attempt 4 would be on tick 7, after the deadline 5
result: timeout

More tasks

  • A breaker and retries together. Put the breaker in front of with_retry. Is a refusal "at once" a temporary error?
  • A bulkhead. Split the pool from the problem in two: 2 threads for payments and 2 for the catalogue. How many catalogue requests are refused now without a timeout?

Test

Check yourself

1. Why is a slow provider more dangerous than a provider that does not work at all?
2. Which errors are worth repeating?
3. Why is jitter added to the wait?
4. The breaker is open. What happens to a new request before the cooldown has passed?
5. Three layers on top of each other, each retries up to 3 times. How many requests can reach the bottom one if it does not answer?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·