epoll vs io_uring vs Kernel Bypass vs FPGA

Every market data system has the same job at the bottom: get bytes from the network card into the program as fast as possible. The methods below are in the order they appeared. Each one removed a cost the previous one couldn’t.

1. Blocking Sockets, One Thread per Connection

The program calls recv() and the thread sleeps until data arrives. To handle many connections, you start one thread per connection.

int fd = socket(AF_INET, SOCK_STREAM, 0);
connect(fd, (sockaddr*)&exchange, sizeof exchange);

char buf[4096];
while (true) {
    ssize_t n = recv(fd, buf, sizeof buf, 0);   // thread sleeps until data arrives
    if (n <= 0) break;
    handle(buf, n);
}

Good: the simplest model to write and read.
Cost: thousands of connections means thousands of threads, each with its own stack, and the CPU spends its time switching between them. Every message also pays for a system call, a copy and a wake-up.

2. select() and poll(): One Thread, Many Sockets

One thread hands the kernel a list of sockets and asks which ones have data. poll() is the same idea without select()’s limit on the number of sockets.

fd_set ready;
while (true) {
    FD_ZERO(&ready);
    for (int fd : sockets) FD_SET(fd, &ready);   // rebuild the full set on every call
    select(max_fd + 1, &ready, nullptr, nullptr, nullptr);
    for (int fd : sockets)                       // then scan every socket again
        if (FD_ISSET(fd, &ready)) handle(fd);
}

Good: no thread per connection.
Cost: on every call, the whole list is copied into the kernel and every socket is checked, even if only one has data. The work grows with the number of sockets, not with the number of messages.

3. epoll: The Kernel Keeps a Ready List

Sockets are registered once. When data arrives on one, the kernel adds it to a ready list, and epoll_wait() returns only those. (BSD and macOS have the same idea as kqueue.)

select and poll check every socket on every call; epoll returns only the ones that are ready.

int ep = epoll_create1(0);
for (int fd : sockets) {                         // register each socket once
    epoll_event ev{};
    ev.events = EPOLLIN;
    ev.data.fd = fd;
    epoll_ctl(ep, EPOLL_CTL_ADD, fd, &ev);
}

epoll_event ready[64];
while (true) {
    int n = epoll_wait(ep, ready, 64, -1);       // returns only sockets with data
    for (int i = 0; i < n; ++i) handle(ready[i].data.fd);
}

Good: the cost depends on how many sockets are ready, not how many exist. That is the key difference in epoll vs select. This made servers with tens of thousands of connections practical, and it is still the default on Linux.
Cost: every message still goes through the kernel’s network stack, a system call and a copy into the program’s memory.
Tuning: pin the network thread to its own CPU core and let it busy poll (spin, instead of sleeping and waiting to be woken). This removes the wake-up delay, at the price of a core running at 100%. See Where the Thread Runs.

4. Kernel Bypass: Onload, ef_vi, DPDK

Take the kernel out of the path entirely. The network card writes packets straight into memory the program can read, and the program polls for them. There is no interrupt, no system call and no copy.

Through the kernel, a packet passes the driver, the TCP/IP stack and a socket buffer before a system call copies it to the program. With kernel bypass it lands in the program’s memory directly. With an FPGA, the work happens on the card itself.

Onload (AMD Solarflare cards) runs existing socket code, epoll included, with the network stack in user space. Often no code change at all:

$ onload ./feed_handler      # same binary, same epoll code; the network stack now runs in user space

ef_vi (Solarflare) and DPDK (many cards) give the program raw packets. Faster still, but you parse the Ethernet, IP and UDP headers yourself and handle anything TCP would have done.

rte_mbuf* pkts[32];
while (true) {                                   // busy poll: this core never sleeps
    uint16_t n = rte_eth_rx_burst(port, queue, pkts, 32);
    for (uint16_t i = 0; i < n; ++i) {
        // raw Ethernet frame: parse the Ethernet, IP and UDP headers yourself
        handle(rte_pktmbuf_mtod(pkts[i], const uint8_t*), pkts[i]->data_len);
        rte_pktmbuf_free(pkts[i]);
    }
}

Good: latency drops from tens of microseconds to a few, or below one. This is where market data and trading systems moved.
Cost: specific network cards, dedicated CPU cores spinning at 100% (set up as in Where the Thread Runs), and the standard Linux tools (firewall, tcpdump, the usual network statistics) no longer see the traffic.

5. FPGAs: Moving the Work Onto the Card

An FPGA is a chip you program as a circuit. Placed on the network card, it can decode the exchange feed, update the order book and decide to send an order, without the packet ever reaching the main CPU. The software on the server only monitors it and changes its settings.

Good: tens to hundreds of nanoseconds from a price change arriving to an order leaving. The fastest that exists.
Cost: the logic is written in a hardware language (Verilog or VHDL, or generated from C++ with high-level synthesis), it takes much longer to change and test, and the hardware and skills are expensive. Worth it only when being first is the business.

6. io_uring: Fast I/O Inside the Kernel

The newest option keeps the kernel but removes most of the system calls. The program and the kernel share two ring buffers: the program writes requests into a submission queue, and the kernel writes results into a completion queue.

io_uring: requests and results pass through two queues in memory shared by the program and the kernel.

io_uring ring;
io_uring_queue_init(256, &ring, 0);

for (int fd : sockets) {                         // queue a receive on every socket
    io_uring_sqe* sqe = io_uring_get_sqe(&ring);
    io_uring_prep_recv(sqe, fd, buffer(fd), BUF_SIZE, 0);
    io_uring_sqe_set_data64(sqe, fd);
}
io_uring_submit(&ring);                          // one system call for all of them

while (true) {
    io_uring_cqe* cqe;
    io_uring_wait_cqe(&ring, &cqe);              // a finished receive
    int fd = (int)io_uring_cqe_get_data64(cqe);
    handle(fd, buffer(fd), cqe->res);            // res = bytes received
    io_uring_cqe_seen(&ring, cqe);
    // queue the next receive on fd; submit in batches
}

Good: many operations per system call, or none at all with kernel-side polling (SQPOLL), whose polling thread needs its own core too (see Where the Thread Runs). Pre-registered buffers and multi-shot receives cut the remaining costs. Higher throughput than epoll, with standard Linux networking and tools still working.
Cost: a more complex programming model, and still the kernel’s network stack, so it doesn’t reach kernel-bypass latency.

Applies to All: Where the Thread Runs

Whichever method reads the network, the thread doing it needs a quiet, predictable place to run. Four settings matter most, and they matter more the faster the method: with kernel bypass, any interruption shows up directly as latency.

Pin the thread to one core. The scheduler can no longer move it, so its data stays in that core’s cache.
Isolate that core. Keep other programs and the kernel’s timer ticks off it, so nothing interrupts the thread.
Move interrupts elsewhere. Point the network card’s interrupts at other cores.
Stay on one NUMA node. Servers with two CPU sockets have memory and devices attached to each. Keep the network card, the thread and its memory on the same socket.

The network card is attached to one CPU socket. A thread on that socket reads packets directly; a thread on the other socket pays an extra hop across the interconnect for every packet.

// Pin this thread to CPU 3 (a logical CPU, counted from 0): isolated, and on the same NUMA node as the network card.
cpu_set_t cpus;
CPU_ZERO(&cpus);
CPU_SET(3, &cpus);
pthread_setaffinity_np(pthread_self(), sizeof cpus, &cpus);

CPU numbers are logical CPUs, not physical cores. Check the layout with lscpu -e: choose a CPU on the network card’s NUMA node, and leave its hyper-thread sibling idle (or turn hyper-threading off), so the pinned thread doesn’t share its core with anything else.

# Which NUMA node is the network card on?
cat /sys/class/net/eth0/device/numa_node

# Kernel boot options: keep other tasks and timer ticks off cores 2-5
isolcpus=2-5 nohz_full=2-5

# Send the card's interrupts to cores 0-1 only (and stop irqbalance from moving them back)
echo 0-1 > /proc/irq/<irq-number>/smp_affinity_list

Good: lower and, more importantly, steadier latency. Pinning and isolation remove most of the occasional slow messages.
Cost: the isolated cores do nothing else, and a wrong NUMA choice quietly undoes the rest of the tuning, so check it on every new server.