Threads in Python: why and when

Problem

The customer waits three times

The customer clicks "Order". Before they see "Thank you", the server has to ask three services over the network: the payment (does the bank accept the card), the stock (is it in stock) and the e-mail (has the confirmation gone out). Each one answers in about 200 ms.

Here the network is Network. call waits 200 ms and returns the service's answer. The program asks the three one after another:

net = Network()
payment = net.call("payment", "accepted")
stock = net.call("stock", "available")
email = net.call("e-mail", "sent")
print(payment)
print(stock)
print(email)
print(f"Rounds of waiting: {net.rounds}")

payment: accepted
stock: available
e-mail: sent
Rounds of waiting: 3

Each question starts after the previous one has finished. The customer waits for the sum: about 600 ms. And the whole time the processor does nothing: it only waits for an answer.

Network also counts the rounds of waiting: a new round starts when the program asks and no other answer is being waited for. Here there are three, one for each question. In this lesson speed is measured with this count and not in milliseconds: time depends on the machine, while the number of rounds is the same on every run.

More precisely, if you are curious

This is how Network works. with self.lock is from the lesson "Threads": the two counters are touched by several threads at once. Why numbers get lost without it is the next lesson, "Race condition".

class Network:
    def __init__(self):
        self.lock = threading.Lock()
        self.in_flight = 0
        self.rounds = 0

    def call(self, service, answer):
        """Asks a service over the network: the answer comes after ~200 ms."""
        with self.lock:
            if self.in_flight == 0:
                self.rounds += 1  # nobody is waiting: a new round
            self.in_flight += 1
        time.sleep(0.2)
        with self.lock:
            self.in_flight -= 1
        return f"{service}: {answer}"

The idea

Work that waits and work that computes

A program's work comes in two kinds.

Work that waits (I/O-bound): a request to the bank, a query to the database, reading a file, sending an e-mail. The program asks and stands still until the answer comes. Meanwhile the processor is free.
Work that computes (CPU-bound): 1000 invoices as PDF, product thumbnails, a report over a million orders. Here the processor is busy the whole time.

A thread that waits does not use the processor. That is why threads help work that waits even on one core: the first thread asks and goes to sleep, the second asks and goes to sleep, so does the third, and the three waits run together. Work that computes needs a core the whole time. It only gets faster with more cores, one per thread.

In Python this is even sharper because of the GIL: at any moment only one thread runs Python code. Waiting releases the GIL (time.sleep, the network, files), so threads help work that waits. Computing holds it, so threads do not speed it up. It needs processes.

So the first question to ask about a slow program is "what is it waiting for?". The answer decides what helps.

See it

All three at once

The three questions become tasks in a pool of 3 threads (ThreadPoolExecutor from the lesson "Threads"). submit returns a Future right away, and result() waits for the answer:

net = Network()
with ThreadPoolExecutor(max_workers=3) as pool:
    payment = pool.submit(net.call, "payment", "accepted")
    stock = pool.submit(net.call, "stock", "available")
    email = pool.submit(net.call, "e-mail", "sent")
    print(payment.result())
    print(stock.result())
    print(email.result())
print(f"Rounds of waiting: {net.rounds}")

payment: accepted
stock: available
e-mail: sent
Rounds of waiting: 1

net = Network()
with ThreadPoolExecutor(max_workers=3) as pool:
    payment = pool.submit(net.call, "payment", "accepted")
    stock = pool.submit(net.call, "stock", "available")
    email = pool.submit(net.call, "e-mail", "sent")
    print(payment.result())
    print(stock.result())
    print(email.result())
print(f"Rounds of waiting: {net.rounds}")

net = Network()
payment = net.call("payment", "accepted")
stock = net.call("stock", "available")
email = net.call("e-mail", "sent")
print(payment)
print(stock)
print(email)
print(f"Rounds of waiting: {net.rounds}")

Step by step, in words

An order needs three answers over the network: the payment, the stock check and the confirmation e-mail. Each one waits about 200 ms.
One after another: the thread asks for the payment and does nothing until the answer comes.
Only then does it ask for the stock check. Another 200 ms of waiting.
Only then does it ask for the e-mail. Another 200 ms of waiting.
3 rounds of waiting: the customer waits about 600 ms, and the processor stands idle the whole time.
Side by side: the three questions go out at once, each on its own thread, and the three waits overlap.
The program waits for the answers. By the time the first one comes, the other two are ready too.
1 round of waiting: about 200 ms, as long as the slowest answer. Same answers, same order, a third of the wait.

The answers are the same and in the same order. There is one round of waiting: all three questions go out before any answer has come. The customer waits about 200 ms, as long as the slowest service, instead of 600.

The threads here hardly use the processor: three short runs of call and three sleeps. That is why this works on a server with one core too.

More precisely, if you are curious

For many questions at once Python also has asyncio: one thread that juggles thousands of waits without a thread for each. That is the lesson "async is not a thread".

Try it

The e-mail waits for the payment

A colleague changes the order: the e-mail has to say whether it is paid and in stock, so it goes out only after the payment and the stock have answered.

net = Network()
with ThreadPoolExecutor(max_workers=3) as pool:
    payment = pool.submit(net.call, "payment", "accepted")
    stock = pool.submit(net.call, "stock", "available")
    # the e-mail says whether it is paid and in stock
    print(payment.result())
    print(stock.result())
    email = pool.submit(net.call, "e-mail", "sent")
    print(email.result())
print(f"Rounds of waiting: {net.rounds}")

Answer

payment: accepted
stock: available
e-mail: sent
Rounds of waiting: 3

payment: accepted
stock: available
e-mail: sent
Rounds of waiting: 1

payment: accepted
stock: available
e-mail: sent
Rounds of waiting: 2

Try it

Wait for the answer

What waits for the payment answer?

payment = pool.submit(net.call, "payment", "accepted")
print(payment.)

result() waits for the task to finish and returns its result.

Problem

A thousand invoices at the end of the month

At the end of the month the shop makes a PDF invoice for each of 1000 orders. Nothing waits for the network here: the processor computes the whole time. render stands in for drawing the PDF with a computation that only loads the processor and returns a checksum:

def render(invoice):
    """Renders the invoice (here: a calculation that only loads the CPU)."""
    h = invoice
    for i in range(2000):
        h = (h * 31 + i) % 1_000_003
    return h


def render_range(bounds):
    """The invoices from start to end, inclusive."""
    start, end = bounds
    return sum(render(n) for n in range(start, end + 1))

total = render_range((1, 1000))
print("Invoices: 1000")
print(f"Checksum: {total}")

Invoices: 1000
Checksum: 499409728

One core computes everything while the others stand idle. There are no rounds of waiting here: nothing waits.

See it

Four processes, four cores

First with threads. The result is correct, but because of the GIL the four threads compute one at a time and get no faster:

bounds = [(p * 250 + 1, p * 250 + 250) for p in range(4)]
with ThreadPoolExecutor(max_workers=4) as pool:
    total = sum(pool.map(render_range, bounds))
print("Invoices: 1000")
print(f"Checksum: {total}")

Invoices: 1000
Checksum: 499409728

With ProcessPoolExecutor each part goes to a separate process: a separate copy of Python with its own GIL, on its own core. pool.map returns the results in the order of the parts. if __name__ == "__main__": keeps the main code from running again in every process:

if __name__ == "__main__":
    bounds = [(p * 250 + 1, p * 250 + 250) for p in range(4)]
    with ProcessPoolExecutor(max_workers=4) as pool:
        total = sum(pool.map(render_range, bounds))
    print("Invoices: 1000")
    print(f"Checksum: {total}")

Invoices: 1000
Checksum: 499409728

The checksum is the same: splitting does not change the result, only who computes it.

How much faster it is depends on the machine. On my laptop, with a computation 25 times heavier per invoice (50,000 steps instead of 2000), one thread took about 1.2 s, four threads still about 1.2 s, and four processes about 0.4 s.

These are example times, not a promise. More parts than cores do not help: they just take turns on the same cores.

More precisely, if you are curious

How many threads. For work that computes, as many as the cores: os.cpu_count() tells you how many. For work that waits, many more is fine: they only sleep.
Not everything splits. If a tenth of the work cannot be split (for example adding up the parts or writing one file), no number of cores makes it more than 10 times faster. That is Amdahl's law.

Python 3.13 also has a build without the GIL (free-threaded). In it threads compute in parallel, but it is still experimental.
Processes do not share memory: their parts are passed in and the result comes back by copying. So you give a process a big chunk of work, not one invoice at a time.

The idea

What helps

	Work that waits	Work that computes
What slows it down	the answer from the network, the disk, the database	the computing itself
What helps	waiting for several things at once	more cores
In Python	threads, asyncio	processes: ProcessPoolExecutor
On one core	helps	does not help

Threads come at a price. As long as each one works with its own data (its own service, its own 250 invoices), all is well. As soon as two threads touch the same thing (a shared counter, the stock in the warehouse), errors start that do not show on every run. That is the next lesson: "Race condition".

Exercise

The product page

Solve the task in your editor.

The product page. Before it shows the page, the server asks three services: the price, the reviews and the delivery. Each answers in about 200 ms and does not depend on the others. Use Network from the first card and ask them all at once, so that there is one round of waiting:

price: 89 €
reviews: 4.6 of 5
delivery: tomorrow
Rounds of waiting: 1

More tasks

Five services. Add stock and a discount. How many rounds are there if the discount depends on the price?
How many cores. Run the invoices with 1, 2, 4 and 8 parts and time them on your machine. When does it stop getting faster? Compare with the number of cores on your machine. - The wrong tool. Put a 200 ms pause/sleep in render for every tenth invoice, as if it waits for the disk. What helps more now: more threads or more cores?

Test

Test yourself

1. An order waits for three services of 200 ms each that do not depend on each other. How long does the customer wait if you ask them all at once?About 600 ms: the sumAbout 200 ms: as long as the slowest oneAbout 67 ms: a third

2. The server has one core. Do threads help work that waits for the network?Only if there are at least as many threads as coresYes: a waiting thread does not use the processor, so the others run while it waitsNo: on one core the threads take turns and it all takes as long as one after another

3. You have to compute 1000 invoices in Python on 4 cores. What speeds it up?ProcessPoolExecutor with 4 processesThreadPoolExecutor with 1000 threadsThreadPoolExecutor with 4 threads

4. Why does the lesson count rounds of waiting and not milliseconds?Because rounds are smaller numbersThe time depends on the machine and its load, while the number of rounds is the same on every runBecause milliseconds cannot be measured

Why threads

Cookies