Introduction
Threads are very expensive - they consume memory (1Mb per thread) and they consume time in their initialization, finalization and context switching. Therefore, threads should be managed carefully by the thread pool. The goal is to create threads no more than needed.
When a thread is executing an I/O operation such as networking, file system, etc., the thread is being blocked by Windows while the hardware device is performing the I/O operation. The thread will continue to run when the device will finish its operation. So far so good since a waiting thread should not waste a precious CPU time.
But there is a problem though - a blocked thread does not return to the thread pool and thus forcing the thread pool to create new threads for incoming requests, or even worse - reject incoming requests.
Not only I/O operations block threads. For example: SqlConnection.Open can block a thread until it will have an available connection to supply.
Consider the following piece of code:
Here we have 4 blocking operations which are highlighted in yellow.
This code is quite problematic for a scalable server. Imagine a server that serves tens, hundreds or even thousands of concurrent requests - this means that while waiting for these operations to finish and thus finally release the thread, new requests are keep coming in and since threads are blocked and are not being released back to the thread pool, the thread pool will produce more and more threads in order to handle the incoming requests. At best, the thread pool will manage to handle all the incoming requests by producing more and more threads which will eventually decrease the performance dramatically (as stated above). At worst, the number of threads will reach the limit of the thread pool and thus, incoming requests will be queued.
Asynchronous Programming
Previous versions of .net prior to 4.5 already had a solution (not completed) to the problem stated above.
The solution was in the form of Asynchronous Programming which includes all the Begin_xxx and End_xxx methods. For example: SqlCommand.BeginExecuteReader, WebRequest.BeginGetResponse and so forth.
Below, is an example code:
figure 2
Async Programming in general is not the objective of this post so I'm not going to explain in details how SqlCommand.BeginExecuteReader works. But i will say that when the thread is invoking BeginExecuteReader it is not blocked and it is free to continue to the next line of execution. When the SqlCommand finishes to execute the query against the DB, the inline method supplied to BeginExecuteReader will be invoked by some available thread, not necessarily the one who called BeginExecuteReader. And thus, no threads are being blocked like they were if they were invoking SqlCommand.ExecuteReader.
As mentioned, this code belongs to .net versions prior to 4.5 and it has some major drawbacks:
1 - using statements can not be used.
2 - Inline methods are not so intuitive.
3 - there are no async executions for DataReader.GetInt32, DataReader.Read and SqlConnection.Open. Therefore, blocking operations are not fully avoided.
Actually, SqlConnection.Open can be a real bottleneck. I made an experiment: I have created an asp.net application with 2 pages: light.aspx and heavy.aspx. The light one should perform a quick and simple operation, say, some simple calculation. The heavy one should perform a heavy and long operation, say, executing some heavy query against the db which might take a few seconds.
The heavy page will use a connection pool (not a thread pool!) of 30 connections.
I have implemented the heavy page in 2 versions: a synchronous version which will implement the code from figure 1 and an asynchronous version which will implement the code from figure 2.
For both versions, a simple client application that i've built, sent 1000 requests for the heavy page and then 1000 requests for the light page.
What I'm interested to see in such an experiment is how the light pages are responding when there are many requests for heavy pages that are consuming threads from the thread pool.
I've expected that the results of the asynchronous version will be better since less threads will be blocked by the heavy page and thus less threads will be created by the thread pool.
I was wrong - at both versions the server became irresponsive at some point for both light and heavy pages.
I expected this for the synchronous version, but why did it also happened for the asynchronous version?
The answer is this: the first 30 requests were not blocked since SqlConnection.Open had available connections to supply them. But from the 31th request and forth, SqlConnection.Open blocked all threads until some of the first 30 threads will finish their job and release their connections. Thus, more and more threads became blocked, hence, increasing the load on the thread pool. At some point, new incoming requests, whether they were for heavy pages or light pages could not be handled and thus where queued.
Now we'll see how .net 4.5 can help us solve this problem.
.Net 4.5 - Asynchronous Programming
In the code below you can see the new way to implement async operations with .net 4.5:
figure 3
The first thing to notice about is two new keywords: await and async. To support these 2 new keywords you have 2 options: upgrade VS 2010 by installing this and then this, or you can start working with higher versions of VS: 2011 or 2012.
But since we are using features of the async ADO.Net which is part of the .net 4.5 - we cannot use VS 2010 which doesn't support them anyway (as far as i know).
OK, now let's analyze it.
In figure 4 you can see the control flow of the thread that will execute the method shown in figure 3. There you can see that thread t1 is the executing thread and it is the one who calls SqlConnect from within Foo.
When the thread is executing the line: await con.OpenAsync(); ado.net is trying to allocate a free and available connection in the connection pool. If all connections are taken (and the limit is reached), the thread will skip all the code below that line and will return back to the point in which it entered SqlConnect and will continue to execute the Foo method. The code below await con.OpenAsync(); will be executed when some connection will become available and it will be executed by a thread which most likely won't be t1 (t2 in figure 4).
figure 4
Of course, the same goes for all the other lines involving the await keyword, meaning that when the new thread (t2), which will execute the rest of the code, will reach a line 99 in figure 3, it will skip all the lines below it and return back to the thread pool.
This ensures us that SqlConnect does not involve any blocking points and thus, no thread will be blocked by SqlConnect and this will increase the overall availability of the thread pool's threads.
I went back to my experiment and changed the heavy page to implement the code from figure 3.
Just to remind you, what I'm interested to see in such an experiment is how the light pages are responding when there are many requests for heavy pages that are consuming threads from the thread pool.
I run my test again and... good news! for the new version, the responsiveness of the server for light pages was the same whether heavy pages were running in the background or whether not. It means that the heavy pages did not add any significant load on the thread pool.
By monitoring the thread pool this came up to be true - the thread pool hardly needed new threads to handle the requests for both heavy and light pages.
