How does NodeJS handle thousands of requests while it’s single-thread?

How does NodeJS handle thousands of requests while it’s single-thread?

10 mins read

Hi, it’s been a long time since the last time I wrote a post on my blog page. Those days were tough for me because I have a lot of things to worry about. But now I’m back with a new post. Note that this is just my opinion/point of view about this (of course I got some knowledge from friends, and blogs)

Many languages use multi-thread models like PHP, Java, and C# but JS still chooses to stay with single thread event loop model because it has some advantages

  • The first thing to get into account here is that Javascript was born with the purpose of building a scripting language for front-end development. Its aim is to build a fast, easy-to-use language to fit into front-end development. Dealing with a single thread is much easier than multi threads where you have to face problems of design architecture (since threads share the same resources), deadlocks, race conditions, and much more things that can make you crazy.
  • Secondly, with the multithreading model, for each request, you’ll create a new thread to handle it. The problem here is that the number of threads you can create depends on the server’s RAM capacity. In single-thread, the thing that you have to worry about is the CPU, not RAM. But with the speed of the modern CPU which can handle millions of operations per second, for application which does not require many CPU-intensive requests, using the single thread model is the best choice (We’ll talk about the reason why we need to care about the CPU intensive requests in the next sections where we discuss how NodeJS handle a request), by not spawning a new thread for each request, it will reduce the memory and resource usage

Before diving into the main part of this section, let’s go through some basic definitions so that when we come to the next part, it would be easier to understand the secret behind NodeJS.

NodeJS is built on top of V8 Engine and LibUV. In this article, the thing we need to care about is LivUB. It’s written in C and contains the Event Loop, Thread Pool (which provides more threads that can be used by NodeJS), and Event Queue. The number of threads in the threads pool in LibUV by default is 4. So basically, NodeJS will have 1 main thread and 4 sub-threads (I don’t know if they can be called “sub-thread”, but just keep in mind that NodeJS does not go around with only 1 thread)

Now, let’s come to one of the biggest arguments of all time among Javascript developers (also some other developer communities). It is: Is NodeJS single-threaded?

  • One of the proofs given by the people who believe that NodeJS is not a single thread is that: NodeJS is built on top of V8 Engine and LibUV, LibUV has the thread pool which contains some other threads that NodeJS can use, so basically, NodeJS has more than 1 thread, therefore, it’s not true that NodeJS is single-threaded. In this argument, I’m on the side of people who think that NodeJS is Single-Threaded. Let me explain why: Yeah, that’s true that NodeJS can take advantage of using threads provided by LibUV. But keep in mind that, the keyword “single-threaded” is about the ability to execute the same piece of code more than 1 at the same time. For example, if you have a loop that logs in the console a word, i.e: “Hello world”. In NodeJS, within an instance of a NodeJS application, there can’t be 2 loops run at the same time simultaneously, you can test it yourself.

CPU bound, IO bound task

  • I/O = input/output task such that: reading some rows from database and returns it, counts number of lines in a file,…
  • CPU: Tasks that your CPU has to do the job. i.e: calculate a operation: 1+1 = 2, image processing.
  • So just remember that the I/O task is something that the CPU can’t handle itself but must have the I/O subsystem do it
  • CPU task must be executed in the main thread of NodeJS, while the I/O task can be executed in threads within the thread pool provided by LibUV. That’s the reason why NodeJS is not suitable for applications with CPU-intensive tasks. We’ll go into detail in the next few minutes.

Blocking I/O vs Non-Blocking I/O:

  • Blocking I/O: Requests involved with I/O operation will block the thread/process until it’s done
  • Non-blocking I/O: Request involved with I/O operation will be pushed into the queue and executed later (if it’s not ready, if it’s ready, returns it right away)

Okay, it’s enough for the definition stuff. Now go to your main concern when you go here.

Here’s how NodeJS handle a request.

When a request comes, it is first pushed into a queue called the event queue.

The event loop will pick up the first element in the event queue and start to process it. There are 2 cases here:

  • If the request is only involved not blocking the IO task / CPU task (do some calculation, no need to call other IO subsystem) (in the above example, it’s the job of calculating the bill of the order), the event loop will push it to the call stack and NodeJS will execute it right away in the main thread, here comes the first bottleneck of NodeJS, when NodeJS is handling that task in the main thread, event loop won’t pick up any new task from the event queue, all other tasks must wait for the processing task to be done which leads to the fact that the response time will be extremely slow. That’s the reason why NodeJS is not suitable for CPU-intensive tasks, CPU tasks can only be done in the main thread, when the main thread is busy, it can't do anything else.
  • If it’s a non-blocking IO task, the event will push that job to another queue, tasks in that queue are waiting for tasks to be done by threads in the threads pool provided by LibUV. Please note that by default, LibUV only provides 4 additional threads to process blocking I/O tasks. That means when these 4 threads are busy handling tasks. Every 5th task that comes in must wait (in the queue I mentioned above). But keep in mind that the number of threads provided by LibUV can be changed via settings (but as far as I know, 4 is enough to handle most requests), if the number of requests is too big, that time we should consider horizontal scaling (create another instance of the application) instead of increasing the number of LibUV’s threads.
    • Some might wonder, how could it be possible that a NodeJS application can handle thousands of requests while it only has 5 threads in total? The point here is “callback”. When a thread (A) in the thread pool picks up a task, it will interact with other I/O subsystems like a database (for example). And note that, the database also provides some threads (B) itself, so thread A just comes in and says “hey, database, I need to get some data in the database, but I won’t wait for you to find and give it back to me, instead I will leave a “callback” here, call it when you’re done with your job, I will go back to pool and pick up another job”. Once the database’s done its job, it will call the callback function and send the response back to the callback queue, and when the call stack is empty, the event loop will pick from the callback queue to the call stack to continue to handle the request.

Below is a simple diagram visualize how NodeJS handle a request

https://www.digitalocean.com/community/tutorials/node-js-architecture-single-threaded-event-loop

Follow-up questions:

You guys might have some doubts about this (like I did), so I will list some questions here, hope that they will (partially) match your concerns.

❓ Okay, I understand what you said in (1). But I don’t get it at all, so what will happens after that, it will send back the response to the call stack, and then what?

→ Yeah, right, keep in mind that your function that handles the request is not usually simple, it might involve with many calls to the database, or any other blocking I/O task. Like this

sql
const handleRequest = async (input: InputDTO) => {
	const dataA = await dbConnection.get(A);
  const dataB = await dbConnection.get(B);
// process the output which combining dataA and dataB or some other stuff associated with those datas.
}
const handleRequest = async (input: InputDTO) => {
	const dataA = await dbConnection.get(A);
  const dataB = await dbConnection.get(B);
// process the output which combining dataA and dataB or some other stuff associated with those datas.
}

So NodeJS will break your handleRequests function into some smaller tasks. For example, get dataA is 1 task, get dataB is another task, and finally processing the output is another task. each task is separately pushed into the event queue and handled individually, (of course it must be in the order that you write in your code).

  • A small note in this example is that, whenever you call the “await” keyword in an async function, what comes after the await keyword will be automatically wrapped in a Promise. and NodeJS will stop executing your function at the point it meets await keyword, it will continue to process that function when it receives the response from that Promise.

When dataA is returned from the callback, NodeJS will continue to get the dataB from the database, the event loop will pick it (if it’s on top of the call stack) and process it just like how it processed to get the dataA. And it will continue that loop of the process until it reaches the end of the handle request function.

❓ What is the priority in the case the call stack is empty and we have tasks from the event queue and callback queue ready at the same time?

  • In my opinion, I think it will take the task from the callback queue first since that task from the callback queue has something to do with a previous request, it’s better to send the response to the previous request instead of processing the new one if both of them are ready. But it’s just my assumption. Maybe NodeJS does not follow that rule.

❓ I think that callback queue contains some smaller type of queue, doesn’t it

  • You’re right, when I say callback queue, I just want to talk about the high level of processing task routine in NodeJS. Indeed, we have more than 1 callback queue. They are: 1) NextTick queue 2) Micro task queue 3) Timers Queue 4) IO callback queue (Requests, File ops, db ops) 5) IO Poll queue 6) Check Phase queue or SetImmediate queue 7) close handlers queue. The event loop will continuously check these callback queues in the order and if any task in that queue is done, it will move that back to the call stack to continue the process (of course in case the call stack is empty).

Okay, above is all the things that I want to share with you guys today. Note that all of them are based on my understanding and some articles I read on the internet, they might not be true 100% (or maybe completely wrong. If there is something wrong with this article. Please feel free to drop a comment and I will go on to discuss more with you guys. Thank you and bye bye

https://viblo.asia/p/ben-trong-nodejs-2-thu-vien-libuv-RQqKLRMOl7z

https://dattp.github.io/2020-04-10-event-loop-in-nodejs/

https://www.digitalocean.com/community/tutorials/node-js-architecture-single-threaded-event-loop

Tagged:#JavaScript#Web Development
0