Skip to content
wannacode

A pool of threads and tasks

In short: A thread pool is a few workers in front of a task queue instead of a new thread per task; in .NET this is the shared pool with Task.Run, Parallel.For and Task.WhenAll.

Problem

A thread for every image

The shop starts a sale. Sellers upload images of their products, and every image needs a thumbnail for the product list. Making a thumbnail is work that computes: it goes through every pixel (see work that waits and work that computes).

The first version starts a thread for every uploaded image:

var threads = new List<Thread>();
foreach (string image in uploads)
{
    var thread = new Thread(() => SaveThumbnail(image));
    thread.Start();
    threads.Add(thread);
}

With 20 images everything works on your machine: "Done: 20 thumbnails". In the first hour of the sale 10,000 come at once.

  • Every thread needs its own stack (usually up to 1 MB) and a place in the operating system. Thousands of threads eat the server's memory.
  • Even if there is memory, 10,000 threads on 8 cores only take turns. The processor wastes time switching them, and every image gets slower.

The server does not slow down; it falls over. And a server that is down does not serve the buyers either.

The idea

A few workers and a queue

A thread pool is a few workers, created once, and a queue of tasks in front of them. A task is "shrink this image". A free worker takes the next task from the queue, finishes it and comes back for a new one.

  • The number of workers is constant. 10,000 images do not make 10,000 threads, but 10,000 tasks in the queue. A task is a small object; a thread is expensive.
  • The workers are reused. A thread is not created and destroyed for every image.
  • The queue absorbs the peak. When uploads come faster than the workers can handle, the tasks wait instead of bringing the server down.

This is producer and consumer: the code that accepts uploads puts tasks into the queue, and the workers are the consumers.

You already met the pool in the lesson on threads: Task.Run, Task<int> and .Result. Here we will see what happens inside, how to order the results, what happens to errors and how many workers you need.

See it

Three workers

Making a thumbnail here is a pause, 2 ms for every KB, and the size comes from the name of the image. That way the output is the same on every run:

// Making a thumbnail: here it is a pause, and the size comes from the name.
static int Thumbnail(string image)
{
    if (image.StartsWith("broken"))
    {
        throw new ArgumentException("corrupt file");
    }
    int kb = 20 + 3 * image.Length;
    Thread.Sleep(2 * kb);
    return kb;
}

Eight images, a pool of three workers:

int[] sizes = new int[images.Length];
var options = new ParallelOptions
{
    MaxDegreeOfParallelism = 3,
};
Parallel.For(0, images.Length, options, i =>
{
    sizes[i] = Thumbnail(images[i]);
});
int total = 0;
for (int i = 0; i < images.Length; i++)
{
    Console.WriteLine($"{images[i]}: {sizes[i]} KB");
    total += sizes[i];
}
Console.WriteLine($"Total: {total} KB");
Step by step, in words
  1. 8 images wait. Parallel.For will use at most 3 threads of the .NET pool.
  2. Parallel.For hands out the indexes. The three threads take the first three images.
  3. Thread 3 finishes first: hat is the third image, and its result goes to the third place. The worker does not stop: it takes the next task.
  4. The queue is empty. Thread 1 is done and has nothing to take, so it waits. The others finish their last images.
  5. Every result is in sizes[i], so the order is the order of the images. Total 379 KB.
string[] images = { "boots.jpg", "jacket.jpg", "hat.jpg", "scarf.jpg", "backpack.jpg", "gloves.jpg", "watch.jpg", "cap.jpg" };
int[] sizes = new int[images.Length];
var options = new ParallelOptions
{
    MaxDegreeOfParallelism = 3,
};
Parallel.For(0, images.Length, options, i =>
{
    sizes[i] = Thumbnail(images[i]);
});
int total = 0;
for (int i = 0; i < images.Length; i++)
{
    Console.WriteLine($"{images[i]}: {sizes[i]} KB");
    total += sizes[i];
}
Console.WriteLine($"Total: {total} KB");
Output
boots.jpg: 47 KB
jacket.jpg: 50 KB
hat.jpg: 41 KB
scarf.jpg: 47 KB
backpack.jpg: 56 KB
gloves.jpg: 50 KB
watch.jpg: 47 KB
cap.jpg: 41 KB
Total: 379 KB
  • .NET has one thread pool (ThreadPool) for the whole program and decides by itself how many threads to keep. You do not create it and you do not close it.
  • Parallel.For runs the body for every index in the pool and waits for all of them. MaxDegreeOfParallelism = 3 says "at most three at once": that is the pool of three.
  • Every task writes into its own place sizes[i], so no lock is needed and the results are in the order of the images.
More precisely, if you are curious

.NET decides by itself how many threads the pool keeps: it starts with as many as there are cores and adds more if tasks sit for a long time without computing. ThreadPool.GetMaxThreads and SetMinThreads tune it, but you rarely need to.

Parallel.ForEach(images, options, image => …) is the same for a collection, and Parallel.ForEachAsync is for tasks with await. The thread that calls Parallel.For works too and waits at the end, so in a web application do not call it in the request's thread.

How it is in the other languages: in Java the pool is an object you make and close (ExecutorService), in Python it is ThreadPoolExecutor, and Node has no ready-made pool of Workers.

Try it

Find the bug

A colleague rewrote the thumbnails with a pool, and the sale did not get faster. Six images take as long as without a pool:

var watch = Stopwatch.StartNew();
var sizes = new List<int>();
foreach (string image in images)
{
    Task<int> size = Task.Run(() => Thumbnail(image));
    sizes.Add(await size);
}
Console.WriteLine($"Took over 0.5 s: {watch.ElapsedMilliseconds > 500}");
Output
Took over 0.5 s: True

Pick the line with the bug.

The idea

Results in input order

Tasks finish in an order you do not know: it depends on the images, on the load, on which worker was free. So never print "in the order they finished" if you want the same output every time.

Two ways give the same result on every run:

  • In input order. You keep the promises in a list in the order of the tasks and wait for them in that order. That is how the code above works. In C# Task.WhenAll gives an array in the order of the tasks, not in the order they finished:
  • A sum or a count. If you only need the total, the order does not matter: 47 + 50 is the same as 50 + 47.
Task<int>[] tasks = images.Select(image => Task.Run(() => Thumbnail(image))).ToArray();
int[] sizes = await Task.WhenAll(tasks);
for (int i = 0; i < images.Length; i++)
{
    Console.WriteLine($"{images[i]}: {sizes[i]} KB");
}
Output
boots.jpg: 47 KB
jacket.jpg: 50 KB
hat.jpg: 41 KB
scarf.jpg: 47 KB

Sometimes you want the results in the order they finished: for a "3 of 8 done" bar, or to show the first ready images. Task.WhenAny gives them that way. Then you only collect things that do not depend on the order:

var running = images.Select(image => Task.Run(() => Thumbnail(image))).ToList();
int count = 0;
int total = 0;
while (running.Count > 0)
{
    Task<int> done = await Task.WhenAny(running);
    running.Remove(done);
    total += await done;
    count++;
}
Console.WriteLine($"Done: {count} of {images.Length}, total: {total} KB");
Output
Done: 8 of 8, total: 379 KB
Task.WhenAny gives the first finished task from the list. You remove it and wait for the next one. With many tasks this is O(n²), which is why .NET 9 added Task.WhenEach, which gives them one by one.

The idea

An error inside a task

One of the images is corrupt, and thumbnail throws an exception. It does not come out in the thread that started the task: that thread has long moved on. The pool catches it and keeps it in the promise of the result. It comes out when you wait for that result:

string[] images = { "boots.jpg", "jacket.jpg", "broken.jpg", "hat.jpg" };
var tasks = images.Select(image => Task.Run(() => Thumbnail(image))).ToList();
for (int i = 0; i < images.Length; i++)
{
    try
    {
        int kb = await tasks[i];
        Console.WriteLine($"{images[i]}: {kb} KB");
    }
    catch (ArgumentException e)
    {
        Console.WriteLine($"{images[i]}: error: {e.Message}");
    }
}
Output
boots.jpg: 47 KB
jacket.jpg: 50 KB
broken.jpg: error: corrupt file
hat.jpg: 41 KB
await throws the exception itself (ArgumentException). .Result would throw it wrapped in an AggregateException. The other images do not suffer.

Try it

Where did the error go

Two of the images are corrupt. What does the program print?

string[] images = { "boots.jpg", "broken-1.jpg", "broken-2.jpg" };
Task<int[]> all = Task.WhenAll(images.Select(image => Task.Run(() => Thumbnail(image))));
try
{
    await all;
}
catch (ArgumentException e)
{
    Console.WriteLine($"Caught: {e.Message}");
}
Console.WriteLine($"Errors in the task: {all.Exception!.InnerExceptions.Count}");
Answer

The idea

How many workers

It depends on what they do:

  • Work that computes (thumbnails, PDFs, compression): as many as the cores. A ninth worker on 8 cores has nowhere to run; it only takes turns with the others.
  • Work that waits (e-mail, payment, database): more. While one worker waits for the network, the core is free for another. If a task waits W ms and computes C ms, about 1 + W / C workers fit on a core.
static int PoolSize(int cores, int waitMs, int computeMs)
{
    return cores * (1 + waitMs / computeMs);
}


Console.WriteLine($"Thumbnails (computing only): {PoolSize(8, 0, 50)}");
Console.WriteLine($"E-mails (90 ms waiting, 10 ms computing): {PoolSize(8, 90, 10)}");
Output
Thumbnails (computing only): 8
E-mails (90 ms waiting, 10 ms computing): 80

The number of cores comes from Environment.ProcessorCount. The formula is a start, not a law: measure. If the e-mails go through one server with a limit of 20 connections, 80 workers will only wait in a queue in front of it.

Try it

Fill in

How many workers for a task that waits waitMs and computes computeMs?

static int PoolSize(int cores, int waitMs, int computeMs)
{
    return cores * (1 + );
}
More precisely, if you are curious

In C# you do not keep threads for work that waits at all: await frees the thread while the request waits for the network (the lesson async and await). That is why a web server serves thousands of requests with a small pool. The resources behind them (database connections) are limited, for example with a SemaphoreSlim.

Exercise

The invoices of the sale

Solve the task in your editor.

After the sale every order needs an invoice. Invoice(amount) makes the PDF (here a pause of 50 ms) and returns the amount with 20% VAT. An order with an amount of 0 or less is invalid, and the function throws an exception with the text "invalid amount".

The orders are 1001 to 1008 with amounts €50, €125, €80, €0, €240, €35, €90 and €15. Make the invoices in a pool of 4 workers. Print them in the order of the orders, and the invalid one with the text of the error. At the end: the sum of the good invoices and how many failed.

Output
Order 1001: €60
Order 1002: €150
Order 1003: €96
Order 1004: error: invalid amount
Order 1005: €288
Order 1006: €42
Order 1007: €108
Order 1008: €18
Total: €762, failed: 1

More tasks

  • A bounded queue. Make the pool keep at most 10 tasks waiting in the queue. What happens to the eleventh? (Channel.CreateBounded or SemaphoreSlim)
  • Progress. Print "3 of 8 done" after every finished invoice, and at the end the ordered list again.
  • A timeout. An invoice that takes over 1 second counts as failed, without stopping the others.

Test

Check yourself

1. Why a pool of a few threads, and not a thread for every uploaded image?
2. A pool of 3 workers shrinks 8 images. The third one finishes first. Where is its size when you read them in list order with get, result, Promise.all or sizes[i]?
3. A task in the pool throws an exception. When do you see it?
4. The server has 8 cores. Every e-mail waits 90 ms for the network and computes for 10 ms. About how many workers keep the cores busy?
5. How many workers for making thumbnails, if it only computes and the server has 8 cores?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·