Go back

Building an Event Loop From Scratch

Published:  at  09:43 AM

Here are eleven lines of Node.js. You have probably written something close to this without thinking about it.

const net = require("net");

const server = net.createServer({}, socket => {
  socket.on("data", buffer => {
    console.log(`received: ${buffer.toString("utf8")}`);
    socket.write(buffer);
  });
});

server.listen(8080, "127.0.0.1", () => {
  console.log("Press Ctrl+C to stop.");
});

It is a TCP echo server. Open ten terminals, point nc 127.0.0.1 8080 at it from all of them, and every connection works. No threads in your code. No await in sight. Just callbacks that fire when something happens.

The question I want to chew on in this post is: what is actually running underneath that?

We are going to build the same thing, in C++, from scratch. By the end you will have a tiny event loop that accepts many clients at once and echoes their data back, from a single thread, with no threads in your code either. The reason for doing this is not that you should write your servers in C++. It is that once you have built the thing yourself, even a toy version, socket.on('data', ...) stops being magic. It becomes “oh, that thing I built.”

A note on platform. The code uses kqueue, which is the macOS and BSD mechanism. If you are on Linux, the syscalls are called epoll_create, epoll_ctl, epoll_wait instead. Different names, same shape. Windows calls its version IOCP and has a slightly different API but the same idea. Everything in this post applies.

Why event loops exist at all

Network I/O is slow. Not slow like a tight CPU loop is slow. Slow like “millions of cycles slow.” If your program calls recv() on a socket and the other side has not sent anything yet, the kernel parks your thread until something arrives. While it is parked, your program does nothing. Not even handle a different client.

So you have two obvious ways out.

One: spawn more threads. One thread per connection, each one allowed to block on its own socket. This works. It also burns memory, costs context switches, and falls over somewhere between a few thousand and a few tens of thousands of connections.

Two: use one thread, but never let it block on a single fd. Ask the kernel “tell me when any of these sockets has something for me,” go to sleep, wake up when there is real work, and run the right callback. That is an event loop.

Node picks option two. So does nginx. So does Redis. So does your browser, sort of.

A short detour: sockets, file descriptors, and the TCP lifecycle

Node hides a lot. Before we can write the C++, we need just enough vocabulary to read it.

Open a file in your program. The operating system has to remember which file you opened, what mode, where the cursor is, who owns it. That bookkeeping lives in the kernel. Your program does not get to touch it directly. Instead, the kernel hands you back a small integer, usually starting at 3 and counting up. That integer is an index into a table of “things this process has open.” It is called a file descriptor, often shortened to fd.

The interesting part: a fd does not have to refer to a file. The same integer-as-index pattern is used for pipes, devices, terminals, and network sockets. When you read “everything is a file” about Unix, this is the part that is literally true. A TCP connection is, from your program’s perspective, just a number you can call read and write on.

A TCP server in raw POSIX goes through five named stages. Here is the picture.

socket()  ->  create an fd that represents "a TCP endpoint, not yet connected"
bind()    ->  tell the kernel which address and port this fd should listen on
listen()  ->  mark the fd as a server; accept incoming connections on my behalf
accept()  ->  block until a client connects; return a brand new fd for that client
recv/send ->  read and write bytes on the per-client fd
close()   ->  done with this fd, return it to the kernel

accept is the one that confuses people the first time. It returns a different fd from the one you called listen on. The original (often called the “listening socket”) stays alive and keeps accepting more clients. Each accepted client gets its own fd. Think of the listening socket as a doorbell and the accepted fds as individual conversations.

Stage 1: the naive blocking echo server

This is the smallest C++ program that does what the Node snippet does. It compiles, it runs, it works. It also has a fatal limitation that motivates the rest of the post.

// includes omitted: <iostream>, <unistd.h>, <arpa/inet.h>

int main()
{
    int server = socket(AF_INET, SOCK_STREAM, 0);

    sockaddr_in address{};
    address.sin_family = AF_INET;
    address.sin_addr.s_addr = INADDR_ANY;
    address.sin_port = htons(8080);

    bind(server, (sockaddr *)&address, sizeof(address));
    listen(server, 1);

    while (true)
    {
        int client = accept(server, nullptr, nullptr);

        char buffer[1024];
        int bytes;

        while ((bytes = recv(client, buffer, sizeof(buffer), 0)) > 0)
        {
            std::cout << "received: ";
            std::cout.write(buffer, bytes);
            std::cout << std::endl;
            send(client, buffer, bytes, 0);
        }

        close(client);
    }
}

There is the lifecycle from the previous section. socket, bind, listen, accept, recv in a loop until the client sends nothing more, send back, close. The outer while (true) means after one client disconnects we wait for the next.

Now imagine two people connecting to this server at the same time. The first one’s nc opens. They type. The server logs and echoes. So far so good. The second person’s nc opens too: the kernel’s TCP backlog quietly accepts their TCP handshake. But the program is stuck inside the inner recv loop, reading from the first client. The next iteration of the outer while (true) never starts, so accept is never called again, and the second client’s bytes sit in a kernel buffer that nobody reads. The first client has the program hostage until they disconnect.

This is not a bug. It is the direct consequence of three blocking calls and one thread. accept blocks until a client arrives. recv blocks until that client sends bytes. While either one is blocked, no other client can be serviced.

If we sketch out where the program spends its time, it looks like this.

time -->
[ accept waits ][ recv waits ][ send ][ recv waits ][ send ][ recv waits ] ...
                client 1 has the whole program until they disconnect

We need to fix two things. We need a way to wait on any fd at once instead of one specific fd. And along the way, we should also clean up the structure so that “make a server” and “do something with each connection” stop being smashed into one function.

Stage 2: the same program, behind a Node-shaped API

The blocking is the real problem, but before we fix it, it helps to reshape the program. The reason will become clear in the next stage.

Here is the rough structure we want. A netServer struct with a listen method that takes a port, a host, and a callback. A free function createServer that takes a per-client callback. The body of main should describe intent, not POSIX.

The socket setup goes into the method, with real error paths instead of the original code’s blind faith.

struct netServer
{
    int fd;
    void listen(const int port, const std::string host, void (*fn)());
};

void netServer::listen(const int port, const std::string host, void (*fn)())
{
    fd = socket(AF_INET, SOCK_STREAM, 0);

    sockaddr_in address{};
    address.sin_family = AF_INET;
    address.sin_port = htons(port);

    if (inet_pton(AF_INET, host.c_str(), &address.sin_addr) != 1)  { close(fd); return; }
    if (bind(fd, (sockaddr *)&address, sizeof(address)) == -1)     { perror("bind"); close(fd); return; }
    if (::listen(fd, 1) == -1)                                     { perror("listen"); close(fd); return; }

    fn();
}

A small sidebar on ::listen(fd, 1). We are inside a method called listen and we want to call the POSIX function also called listen. Without the leading ::, the compiler resolves listen to “the method I am currently inside,” which is infinite recursion. The :: says “the one in the global namespace, please.”

main now looks like the Node snippet. The accept/recv/send loop is still the same blocking code as before, but it has been moved out of the server setup into its own block in main.

int main()
{
    netServer server = createServer(onSocket);

    server.listen(8080, "127.0.0.1", onServerListen);

    while (true)
    {
        int client = accept(server.fd, nullptr, nullptr);

        char buffer[1024];
        int bytes;
        while ((bytes = recv(client, buffer, sizeof(buffer), 0)) > 0)
        {
            std::cout << "received: ";
            std::cout.write(buffer, bytes);
            std::cout << std::endl;
            send(client, buffer, bytes, 0);
        }
        close(client);
    }
}

Functionally we are back where we started. The two-client problem is still there. That is the point. The limitation is now isolated to that one while (true) block in main. Everything outside it is reusable. When we swap that block out for an event loop in the next stage, nothing else needs to move.

The two calls keeping us from serving multiple clients have names: accept(server.fd, ...) blocks until somebody new connects, and recv(client, ...) blocks until that one client sends. The reason they are one problem and not two is that they share a root cause. Each one blocks the whole thread until its own specific fd has something for it. We want to wait on any fd at once. That is a different shape of wait.

Stage 3: ask the kernel to tap you on the shoulder

Polling versus notification. Imagine you are waiting for an email. Two strategies.

One: open your inbox every five seconds and check. Every five seconds you stop whatever you were doing, even when there is nothing new. You burn cycles for nothing.

Two: leave the inbox closed. The phone makes a sound when an email arrives. Until then, do other work or sleep. When the sound happens, you know there is something to read.

recv blocking on a single fd is neither of these. It is just stuck, waiting for one specific thing. What we want is the notification model, but for many fds at once. Hand the kernel a list of fds we care about. The kernel watches them on our behalf. When at least one is ready, the kernel wakes us up and tells us which ones.

On macOS, the mechanism for this is kqueue. There are two pieces.

First, a queue object. You ask the kernel for one with kqueue() and get back, yes, another file descriptor. This fd is your handle on a kernel-side list of “things I want to be notified about.”

Second, registrations and waits. You modify the list with kevent() and a struct that says “fd N, please notify me when it is readable.” You wait for events with another call to the same kevent(), this time asking it to fill in an array with whatever is currently ready.

[ user program ]                          [ kernel ]
   kqueue()              -->   creates an empty queue, returns fd
   kevent(q, ADD fd N)   -->   registers fd N for readability
   kevent(q, wait)       -->   blocks until any registered fd is ready,
                                then fills array with the ready ones

The trick is not that we got rid of blocking. We did not. The trick is that the blocking call now blocks on a set of fds. That single shift is the entire idea.

In code, here is the registration. We add it to the end of netServer::listen, after the socket is bound and listening:

kqueueFd = kqueue();
if (kqueueFd == -1) { perror("kqueue"); close(fd); return; }

struct kevent event;
EV_SET(&event, fd, EVFILT_READ, EV_ADD, 0, 0, nullptr);

if (kevent(kqueueFd, &event, 1, nullptr, 0, nullptr) == -1)
{
    perror("kevent"); close(kqueueFd); close(fd); return;
}

EV_SET is a macro that fills in a struct kevent. Read it as “I care about this fd, for readable events, please add it to the queue.” The follow-up kevent call submits that registration to the kernel.

And here is the new main loop. The blocking accept is gone. In its place is a blocking kevent:

while (true)
{
    struct kevent events[10];
    int count = kevent(server.kqueueFd, nullptr, 0, events, 10, nullptr);
    if (count == -1) { perror("kevent"); break; }

    for (int i = 0; i < count; ++i)
    {
        if (events[i].ident == static_cast<uintptr_t>(server.fd))
        {
            int client = accept(server.fd, nullptr, nullptr);
            if (client == -1) { perror("accept"); continue; }

            std::cout << "new connection" << std::endl;
            close(client);
        }
    }
}

kevent returns when at least one registered fd is ready. The count is how many. The inner loop dispatches them. For now there is only one fd registered (the server), so the only event we ever see is “a new client wants in.” We accept and immediately close because we have not built the rest yet.

This version has the right shape but no echoing. It accepts as many simultaneous connections as you want, each of them sees an immediate EOF. We have lost a feature (echoing) to gain a property (concurrency). The next stage puts the feature back.

Stage 4: a real reactor

The piece we are missing is doing something useful with each connection on the same single thread. That means three things: registering every accepted client with kqueue (so the kernel notifies us when they send bytes), routing those notifications to a “data arrived” callback, and cleaning up when a client disconnects.

A watcher is “an fd we care about.” Each watcher carries its own kqueue, plus a flag telling us whether the fd is the server’s listening socket or a client connection. This is a small design shift from the previous stage: the kqueue is no longer a field on netServer. It moves down into each watcher.

struct watcher
{
    int fd;
    int kqueueFd;
    bool isServer;
};

std::vector<watcher> watchers;

bool addWatcher(int fd, bool isServer)
{
    int kqueueFd = kqueue();
    if (kqueueFd == -1) { perror("kqueue"); return false; }

    struct kevent event;
    EV_SET(&event, fd, EVFILT_READ, EV_ADD, 0, 0, nullptr);

    if (kevent(kqueueFd, &event, 1, nullptr, 0, nullptr) == -1)
    {
        perror("kevent"); close(kqueueFd); return false;
    }

    watchers.push_back(watcher{fd, kqueueFd, isServer});
    return true;
}

listen() calls addWatcher once for the server fd with isServer = true. Every time a client connects, we call addWatcher for the new client fd with isServer = false. Each call creates a fresh kqueue.

The three callbacks are the Node events, rewritten as plain functions.

void onSocket(int fd)
{
    addWatcher(fd, false);
}

void onSocketData(int fd, const char *bytes, ssize_t length)
{
    std::cout << "received: ";
    std::cout.write(bytes, length);
    std::cout << std::endl;

    if (write(fd, bytes, length) == -1) { perror("write"); }
}

void onSocketClose(int fd)
{
    for (auto it = watchers.begin(); it != watchers.end(); ++it)
    {
        if (it->fd == fd) { watchers.erase(it); break; }
    }
    close(fd);
}

Map these onto the Node API:

And here is the heart of the program. The outer loop sweeps through every watcher on each pass, asks its kqueue what is ready right now, and dispatches.

while (true)
{
    for (int w = 0; w < static_cast<int>(watchers.size()); ++w)
    {
        watcher watcher = watchers[w];

        struct kevent events[10];
        struct timespec timeout{};
        int count = kevent(watcher.kqueueFd, nullptr, 0, events, 10, &timeout);
        if (count == -1) { perror("kevent"); break; }

        for (int i = 0; i < count; ++i)
        {
            if (events[i].ident != static_cast<uintptr_t>(watcher.fd)) { continue; }

            if (!watcher.isServer)
            {
                char bytes[1024];
                ssize_t length = read(watcher.fd, bytes, sizeof(bytes));
                if (length == -1) { perror("read"); break; }

                if (length == 0)
                {
                    onSocketClose(watcher.fd);
                    break;
                }

                onSocketData(watcher.fd, bytes, length);

                if (events[i].flags & EV_EOF)
                {
                    onSocketClose(watcher.fd);
                }

                break;
            }

            int client = accept(watcher.fd, nullptr, nullptr);
            if (client == -1) { perror("accept"); break; }

            onSocket(client);
            break;
        }
    }
}

One detail to flag in that kevent call: the struct timespec timeout{} is zero-initialised, which makes the call non-blocking. We ask kqueue “anything ready on this fd right now?” and move on either way. The outer loop then visits the next watcher and asks again. The isServer flag on the watcher tells us which branch to take when something is ready.

Three possible paths per event.

  1. Event on a server watcher. A client is waiting to be accepted. accept returns a new client fd, onSocket registers it as a brand-new watcher with its own kqueue, and the outer sweep will visit it next pass.

  2. Event on a client watcher, with bytes. read returned a positive number. We hand those bytes to onSocketData, which echoes them back.

  3. Event on a client watcher, no bytes (or EV_EOF set). read returned 0, or the kernel told us via EV_EOF that the other side closed. We call onSocketClose, which removes the watcher and closes the fd.

There is no accept blocking us. There is no recv blocking us. Each watcher has its own kqueue, each one is asked “anything ready?” in turn, and the program flows on regardless of which clients are silent and which are talking.

This is the moment that made me grin the first time I got it working. Open two terminals, run nc 127.0.0.1 8080 in each, type into them in any order: both echo. Open ten terminals. Same thing. One thread. One outer while loop. No threads, no select, no per-client recv blocking. The same program that could only hold one conversation in stage 1 is now juggling everyone.

The shape we just wrote is the shape that Node, nginx, libevent, libuv, asyncio, and Tokio all share at their core. One loop, one event source, one dispatch into user code. Everything else is detail.

What real event loops do that this one doesn’t

I want to be honest about what we built before closing.

None of these are conceptually hard. They are just the work that turns a 200-line toy into a production runtime. If you ever want to keep going, the file to read in libuv is src/unix/kqueue.c on macOS or src/unix/linux-core.c on Linux. The Bert Belder talks on the Node event loop are the canonical version of this whole story.

Closing

“Event loop” gets used as a magic phrase. Every Node tutorial mentions it. Almost none of them show you what it is. After this post you have built one. It is small, it is macOS-only, and it does not do half of what libuv does. It is still, in every meaningful sense, an event loop.

Next time you write socket.on('data', ...) in Node, you know exactly what is listening for that event. It is a while loop. It is a kevent() call. It is a dispatch into a callback. It was always there.

Share this post on:
Share this post via WhatsApp Share this post on Facebook Share this post on X Share this post via Telegram Share this post on Pinterest Share this post via email
Next Post
Learning C++ by Implementing SOCKS5