Scheduler overhead
The operating system’s scheduler is responsible for choosing which thread should
be executed on which CPU, and when. Even when threads are blocked, the sched-
uler must consider them, to find whether they’re become unblocked. This slows
down context switches, and can slow the entire system.
Between them, these overheads can add to the load on your server, increasing latency
and decreasing throughput.
Remember: the main characteristic of asynchronous code is that the thread that started
a long-running operation is released to do other things. In the case of ASP.NET code,
this thread is from the thread pool, so it is returned to the thread pool during the long-
running operation. It can then process other requests, so fewer threads are needed to
process the same number of requests.
Another Analogy: The Restaurant Kitchen
A web server is a close model of a restaurant. Many customers order food, and the
kitchen tries to satisfy them as soon as it can.
Our kitchen has many chefs, with each chef representing a thread. They cook the dishes
that the customers order, but at points during the preparation, each dish just needs to
be in the oven for a while, and the chef has nothing to do. This mirrors the way that
web requests usually need to make a database query that the web server has no part in.
In a blocking implementation of the kitchen, the chef will sit in front of the oven, waiting
for the dish to be cooked. To model a thread exactly, these chefs have an odd contract
where they aren’t paid while they are waiting for food to cook, because a thread doesn’t
use CPU time when it is blocked. Maybe they read a newspaper.
But even if we don’t have to pay them, and we can hire a new chef for every dish we
need to cook, waiting chefs still take up space in the kitchen. We can’t fit more than a
few tens of chefs in the kitchen before it becomes hard to move around, and everyone’s
work slows down.
Of course, the asynchronous system works much better. Each time food is put in the
oven, the chef notes down what dish it is, and what stage of preparation it’s at, then
finds a new task to do. When the time in the oven is done, any chef can pick the dish
up and continue preparing it.
It’s this efficient system that’s so powerful in web servers. Only a few threads can man-
age a number of simultaneous requests that would have required hundreds before, or
would have just been unfeasible because of the overheads. In fact, some web frame-
works, notably node.js, reject the idea of multiple threads altogether, opting to use a
single thread to process all the requests asynchronously. They can often handle more
requests with one thread than a multithreaded, but blocking, system can handle in
total. In the same way, one well-organized chef in an empty kitchen can cook more
8 | Chapter 2: Why Programs Need to Be Asynchronous