Systems and hardwareDive 05 of 06
What a single read() costs
Follow one HTTP request through a privilege boundary, a sleeping thread, epoll’s ready list and io_uring’s shared rings. Three ways to serve 10,000 connections, down to registers and the packet that wakes a core.
- Levels
- 8
- Reading
- 38 min
- Depth
- Idea to Silicon
- Published
One request · socket 9 · 10,000 open connections
Your code asks.
The kernel owns the bytes.
read(9, buf, 512)Give me the next part of this HTTP request.socket receive queueCopy available bytes, wait for a packet, or return an error.A syscall changes privilege. A context switch changes which thread runs. A read can do the first without doing the second.
Three server designs · same request model
Syscalls per request · common scale 0 to 3
10,000 requests: 20,000 / 20,313 / 626 syscalls.
Model: accepted sockets; one successful read and write; one epoll wait per batch; two io_uring submission waves. Setup, retries and extra waits are excluded.
Ten thousand connections, one request
You have a tiny HTTP server and 10,000 clients connected to it. Most clients are waiting. One sends GET / HTTP/1.1, followed by a header terminator. Your server reads that request and replies with three bytes, ok\n. There is almost no computation to optimise. The interesting question is how the server waits, and how much work it does when waiting ends.
Start with one thread per connection. A handler calls read(9, buf, 512). If the request has arrived, the call copies bytes and returns. If it has not, the handler sleeps inside the kernel. The code looks wonderfully direct: read, parse, write, repeat. Multiply that simplicity by 10,000 and you acquire 10,000 stacks, 10,000 schedulable tasks and a large number of independently arriving wakeups.
An event loop changes the unit of waiting. One thread asks which of its 10,000 sockets can make progress, then reads only those sockets. Linux’s epoll supplies that readiness information. The handler still calls read and write; the expensive collection of sleeping threads has become a collection of connection records. An idle connection needs state, but it does not need to own a thread.
io_uring changes the question again. Instead of asking whether a read would work and then calling read, the server submits a description of the read itself. Later it collects a result. Descriptions and results live in memory shared with the kernel, so several operations can cross the interface in one batch. These are three ways to organise the same request, rather than three different network protocols.
What is the boundary protecting?
Your program runs with restricted privileges. It can execute arithmetic, access its mapped memory, and call functions in its libraries. It cannot tell a network device to send arbitrary host memory or choose another process’s address mappings. The kernel owns those decisions. A A defined entry into the operating system’s privileged code, selected by a number and supplied with arguments in an architecture-specific ABI. asks it to perform an operation under the caller’s permissions and resource limits.
A normal function call remains in the same privilege domain. It changes the instruction pointer and follows a calling convention. A syscall enters a different privilege domain through an entry point the kernel configured. The CPU and kernel cooperate to save enough state, establish a safe execution environment, validate arguments, perform the operation and return. A call to libc can contain such a crossing, several crossings, or none at all.
Local elapsed time · median of 5 runs · ns/call · scale 0–300
Core i7-1355U · CPU 0 affinity · Linux 7.0.0-38-generic · GCC -O2 · 200,000 iterations per run · CLOCK_MONOTONIC_RAW around each loop.
The median here is 1.08 ns for a non-inlined local function, 165.55 ns for glibc getpid(), 154.27 ns for a raw getpidinstruction sequence, and 250.11 ns for a one-byte read from/dev/zero. These are elapsed loop averages, including loop and measurement overhead. CPU affinity pins each run to logical CPU 0; frequency scaling, interrupts and contention still vary. The five raw-getpid averages span 146.28–174.01 ns. There is no universal “syscall costs 150 ns” constant.
The zero device makes the read repeatable without waiting for a network packet or storage. It still performs descriptor lookup, dispatch and a copy to user memory. A socket read adds different work; a blocking socket read can spend milliseconds waiting. Separate CPU service time from elapsed response time before interpreting a profile. Removing a 150 ns crossing will not remove a 20 ms network wait.
Our semantic target is Linux 7.3.4, listed as stable on 15 November 2026. Measurements use the host’s 7.0.0-38-generic kernel. Keep those labels separate: the stable source explains the interface, while the local machine supplies the numbers. We will follow the request downward until the incoming packet makes its bytes available.
A syscall changes privilege. Blocking decides who waits. Batching decides how often you pay for the interface.
A function call with a boundary inside
The handler’s C call has three arguments: an integer descriptor, a pointer to writable memory, and a maximum byte count. The descriptor is a process-local handle. Number 9 does not name a network card or carry a global meaning; Linux uses it to find an open file description. A socket is one kind of object behind that file interface. The WebSockets stream dive explains why those bytes need not align with messages.
On native x86-64 Linux, the syscall number goes in rax. The six argument registers are rdi, rsi, rdx, r10, r8, r9, in that order. Our read uses number 0, descriptor 9, a buffer address and count 512. Argument four uses r10 because syscall uses rcx for the return address. The instruction also overwritesr11 with saved flags. Inline assembly must declare both clobbers and account for memory accessed by the kernel.
AArch64 uses another ABI: svc #0 invokes the supervisor, x8 selects the call, x0 through x5 hold arguments, and x0 receives the result. Native AArch64 read is number 63, not x86-64’s 0. An architecture-independent C function hides those differences. The decoder below deliberately accepts only native x86-64 numbers, excluding the x32 and legacy 32-bit ABIs.
On this entry path, only EAX’s low 32 bits select the syscall. Linux sign-extends them before dispatch, so rax=0x100000027selects getpid, just like 39. The decoder followsentry_64.Sand do_syscall_64; it preserves the x32 ABI bit, 0x40000000, and excludes that separate ABI rather than treating it as a native alias.
read(fd=9, buf=0x1000, count=512)
Linux 7.3.4 native x86-64 ABI, 11 calls. Pointers are displayed, never dereferenced. Integer arguments use their declared ABI widths; count and sigsz are 64-bit.
A pointer in a register is still a user virtual address. Crossing the boundary does not turn it into trusted memory. Linux checks access and uses user-copy machinery; inaccessible memory can produce EFAULT. A count is an upper bound, not a promise that the kernel will fill that much. read may return fewer bytes, return zero at end of stream, or fail. A positive short read is progress, and the caller must keep its parser state.
In the classic x86-64 entry path, entry_SYSCALL_64 saves state and calls the C entry machinery. In Linux 7.3.4, the native dispatch lives in arch/x86/entry/syscall_64.c. It generates switch cases from the syscall definitions. The source retains sys_call_table for tracing; this version does not use it as the native call dispatcher. A diagram that always draws an indirect call through that table is describing an older implementation.
The read wrapper eventually reaches ksys_read and vfs_read in fs/read_write.c. VFS supplies the common file interface, including access checks and accounting. A socket’s read-iterator operation leads into socket code and then the selected transport. For our TCP connection the receive path reaches tcp_recvmsg. Reading a regular file shares the upper interface but reaches a filesystem rather than TCP.
The raw syscall result uses negative error numbers. A nonblocking socket with nothing available commonly returns -EAGAIN, numerically -11 on this ABI. The libc contract converts an error into return value -1 and sets thread-local errno to the positive error number. A successful result does not clear an old errno. Test the return value first, then inspect errno only when the API says it is meaningful.io_uring completions later keep the negative-number convention directly.
The blocking flag belongs to the open file description’s status flags. A descriptor obtained through duplication can share those flags with another descriptor, which matters when a library unexpectedly changes O_NONBLOCK. Per-call flags on APIs such as recv can provide different control. A connection record should therefore know which API contract its owner established, rather than infer it from the fact that the descriptor represents a socket.
Signals also belong in the contract. A read interrupted before transferring bytes can return EINTR, or be restarted under the applicable signal policy. If it already transferred bytes, a short successful result can take precedence. Retrying only the uncompleted portion preserves progress. Blindly restarting a whole higher-level operation can duplicate a write or discard parser state, even though retrying an interrupted primitive seems harmless.
A library function that stays outside
The vDSOis a small ELF image the kernel maps into user space. Supported time queries such as common clock_gettime calls can execute its code, read kernel-published timekeeping data and read a suitable clock counter without entering the kernel. The library chooses that fast path when available; unsupported cases fall back to a syscall. This is possible because the kernel can safely publish the information needed to compute the answer.
A socket read cannot use the same trick. It changes shared socket state and consumes bytes under kernel control. Publishing a timestamp is much simpler than letting arbitrary user code mutate a receive queue. Shared rings will reduce the cost of describing that operation, but they will preserve kernel ownership of the socket itself.
The wrapper, the instruction and the kernel handler are different layers. A function name alone does not tell you how many boundaries were crossed.
Where the thread goes while it waits
Suppose connection 9 has no request bytes yet. The server gave the socket blocking semantics, so returning immediately with EAGAIN would break the contract. The kernel must arrange for this thread to stop consuming a core until useful progress or an interruption becomes possible. It needs a place to remember the waiter and a way to make the waiter runnable again.
That place is a wait queue associated with the socket. The receive path checks its condition, registers a waiter, sets a sleep state and checks again around the sleeping protocol. This ordering closes a critical race: data arriving after the first empty check must either be seen by the recheck or trigger a wakeup of the registered waiter. “Check empty, then sleep” without that handshake could sleep forever beside a full buffer.
A normal interruptible wait uses TASK_INTERRUPTIBLE. Signals can interrupt it, subject to syscall restart rules and progress already made. A socket can also wake because the peer closed, an error occurred, or a timeout expired. Waking means “recheck your condition”; it does not promise that another thread has not consumed the bytes. The loop in tcp_recvmsg and the socket wait helpers implement that distinction.
When the thread schedules away, Linux saves its execution state and chooses another runnable task. The thread is still inside the original read. It has not returned a result and issued a second syscall just because another task ran. When selected again, it resumes the kernel receive path, rechecks, and eventually copies bytes and returns to user mode.
Logical events, left to right · widths represent event slots, not elapsed time
- A runs on the core and calls read on an empty blocking socket. B is runnable.
- A joins the wait queue, rechecks the condition, and sleeps. The scheduler switches to B.
- B uses the core while A waits. A’s read syscall has not returned.
- A packet wakes A. A is runnable; B is still running in this chosen schedule.
- B yields; the scheduler selects A. A resumes its read and copies the bytes.
- A returns to user mode and writes the response. Two task switches occurred: A to B, then B to A.
The timeline has two illustrated context switches: A to B when A blocks, then B to A after the packet arrives and B yields. The syscall entry and return are two privilege transitions but no additional task switches. A ready read can cross into the kernel and return on the same thread without any task switch. An empty read can remain suspended through many intervening task switches.
Who gets the core next?
Linux began moving its fair scheduling class to EEVDF in version 6.6. Earliest Eligible Virtual Deadline First separates eligibility from preference. Lag tracks whether a task is owed service relative to its weight. Among eligible tasks, the scheduler favours the earliest virtual deadline, derived from requested service. Wakeups, preemption rules, affinity and other scheduling classes also matter. Real-time and deadline tasks follow their own policies.
Virtual time is accounting time, not the packet’s wall-clock deadline. A sleeping connection is not continuously consuming its fair share, and a newly runnable handler need not immediately preempt the current task. The interactive above chooses a legal schedule to expose the waiting protocol; it does not calculate EEVDF decisions. To investigate a real delay, distinguish time waiting for network input from time runnable but waiting for CPU service.
One thread per connection retains ordinary control flow across every wait. Its stack remembers local variables and the return path. That is valuable, but it scales memory and kernel scheduling state with the number of connections. On a system configured with an 8 MiB pthread stack reservation, 10,000 handlers reserve 78.125 GiB of virtual stack space before guard pages and other mappings. Reservation is not resident RAM: untouched pages may have no physical backing.
| State budget | Per connection | 10,000 connections |
|---|---|---|
| Example pthread stack reservation | 8 MiB | 78.125 GiB virtual |
| Example event-loop record | 128 B | 1.22 MiB |
| Request buffer, any design | 4 KiB | 39.06 MiB |
| Socket buffers and kernel state | Workload/configuration dependent | Remain in every design |
The record and buffer sizes in this table are design choices. They are not measured kernel object sizes. Reducing pthread stack reservations can make the threaded design practical for more clients, but then stack depth becomes a tighter constraint. The memory dive separates mappings, page faults and resident memory; the Go stack dive explains how growable goroutine stacks change the tradeoff.
Waiting threads do not all burn CPU in a loop. The pressure appears when many become runnable, when their stacks and working sets displace each other’s cache lines, or when thread creation and destruction enter the request path. The number of connections is a capacity question; the number of runnable handlers and their work is a CPU scheduling question. An event loop addresses the first without making the second disappear.
Sleeping retains a continuation. A thread stores it in its execution state and stack; an event loop stores it in a connection record and an explicit state machine.
Ask which sockets can make progress
Replace 10,000 blocking handlers with one loop. Each socket is nonblocking. A read either makes immediate progress or returns EAGAIN, so one slow client cannot strand the loop’s only thread. The loop now needs an efficient answer to “which connections should I try?” Asking read on every socket repeatedly would spend most of its time discovering that nothing changed.
select and poll let the caller pass collections of descriptors and wait for readiness. They rebuild the input set on each call and inspect that supplied collection. Traditional select also uses a fixed-size fd set in the C interface, commonly limited by FD_SETSIZE. Poll uses an array and avoids that particular bitset limit. Both interfaces can be useful for small sets; a large mostly idle population makes repeated scanning costly.
An epoll instance persists its interest set across waits. epoll_ctl(ADD) installs a watch and hooks into the target’s polling wait queues. Socket activity can then run ep_poll_callback and queue a reference to that watch. epoll_wait visits ready candidates, rechecks readiness and copies events to user memory. The relevant machinery lives in fs/eventpoll.c. Registration uses a search structure, and ready delivery has work proportional to the candidates and events handled; “epoll is O(1)” hides too much.
Each returned event includes your saved epoll_event.data. Put a connection ID there and use it to retrieve the parser, buffer and output offset. For the selected HTTP request, read until the parser has enough input, build a response, and write as much as the socket accepts. On EAGAIN, retain the output offset and ask for EPOLLOUT. Stop watching EPOLLOUT when there is nothing left to send, because a healthy idle socket is often continuously writable.
Level and edge triggering
With level triggering, a socket that remains readable can be reported again on a subsequent wait. With edge triggering, delivering an event does not promise a second event merely because you left old bytes unread. New data can generate further notifications even if the queue was already nonempty. ET is not a universal “empty to nonempty only” rule for every file type. Correct code reasons about exhaustion and wakeups, rather than counting edges as messages.
Deliver a chunk, wait, read half, then wait again. LT reports the remaining bytes.
Subset: one nonblocking stream socket, EPOLLIN, one waiter, no EOF/errors. New chunks arrive only into an empty queue. Changing flags starts a fresh socket.
The reliable ET pattern is to use nonblocking descriptors and continue handling readable work until EAGAIN or an equivalent known exhaustion condition. If fairness requires stopping earlier, keep the connection on an application ready queue and resume it without waiting for a new kernel notification. A fast client must not monopolise the loop, but a fairness budget must not accidentally forget unfinished work.
EPOLLONESHOT adds ownership control. Once epoll delivers an event, that watch is disabled until epoll_ctl(MOD) rearms it. A worker can handle the socket without another worker immediately receiving the same watch. Rearming rechecks present readiness, so unread bytes can produce a new event even under ET. This is separate from draining: draining manages buffered work; rearming enables notification again. The simulator lets you forget either action and observe the consequence.
Readiness is a hint to try an operation, not an allocation of bytes to a handler. Another reader can consume the data before your read. The descriptor can also be ready because of EOF or error. Real code handles zero, partial reads, EAGAIN, EINTR and other errors. A socket carrying the last buffered bytes can have a hangup indication at the same time. Throwing away its buffer merely because a hangup flag appeared loses data.
An event loop needs a CPU budget as well as an I/O budget. Parsing a large header, compressing a reply or doing synchronous filesystem work in the callback can block every other connection on that loop. Move substantial CPU work to bounded workers or split it into resumable pieces. The readiness interface solves waiting for I/O; it does not automatically schedule application computation fairly.
Output state is especially easy to lose. If write accepts only 20 of a 60-byte reply, the next attempt must start at byte 20 and request the remaining 40 bytes. Keeping the whole reply and a sent offset is enough for this server. With many clients, enforce an output-buffer limit so slow readers cannot accumulate unbounded queued responses. Backpressure is part of the connection state machine, alongside readiness and parsing.
The herd depends on the topology
Several threads waiting on one shared epoll fd differ from several epoll instances watching the same listening socket. EPOLLEXCLUSIVE addresses wakeup fan-out across instances. The kernel can select one or more exclusive watchers rather than waking every such instance. It does not promise exactly one event, round-robin worker assignment or a global lock on accepting connections.
| Configuration | Meaning |
|---|---|
| One epoll fd, several waiting threads, ET | A readiness wakeup wakes one waiting thread in the documented shared-instance case. |
| Several epoll fds, ordinary watches | The same target wakeup can fan out to all interested instances. |
| Several epoll fds, exclusive watches | One or more exclusive instances receive the wakeup; ordinary watches still receive it. |
| EXCLUSIVE + ONESHOT | Invalid: EPOLL_CTL_ADD returns EINVAL. |
| MOD an EXCLUSIVE watch | Invalid: remove and add a new watch instead. |
The allowed EXCLUSIVE mask is restricted; consult epoll_ctl(2) before combining flags. An exclusive watch can combine EPOLLIN, EPOLLOUT, EPOLLET and EPOLLWAKEUP under the documented restrictions; error and hangup reporting still apply. That is why the one-socket simulator exposes ONESHOT but does not pretend an EXCLUSIVE checkbox could change its single-waiter topology.
epoll reports an opportunity. Your read or write still performs the operation, and your state machine owns incomplete work.
Submit work, collect completions
The loop now scales its waiting state, but each successful request still asks the kernel to read and write separately. io_uring replaces individual calls with records in shared memory. The submission queue, SQ, carries operation descriptions toward the kernel. The completion queue, CQ, carries results back. An operation can remain in flight while the application prepares work for other connections.
Setup begins with io_uring_setup. Its returned parameters describe ring sizes and byte offsets; applications must use those offsets rather than assume a C struct for the entire mapping. The classic arrangement maps an SQ ring, a CQ ring and an SQE array. The SQ ring contains indices into that SQE array. Some setup flags change this arrangement, so a raw implementation must match the flags it requested.
The default submission record, an SQE, is 64 bytes. The default completion record, a CQE, is 16 bytes. Larger layouts and newer modes exist, but the widget below uses the classic layout with an index array, no SQPOLL and one issuer. Its four-entry CQ is explicitly chosen rather than inferred from the default CQ sizing. Operations use separate synthetic sockets and buffers, avoiding overlapping reads on one TCP stream.
CQ head 0 · tail 0
Submission queue · 4 slots
Completion queue · 4 slots
Prepare a read. Its bytes exist before the kernel can see it; release the SQ tail to publish it.
Last prepared SQE · 64 bytes
00 16 00 00 00 09 00 00 00 08 00 00 00 00 00 00 00 00 16 00 10 00 00 00 00 00 00 24 00 02 00 00 00 00 00 00 32 01 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00 00 48 00 00 00 00 00 00 00 00 56 00 00 00 00 00 00 00 00
CQE · 16 bytes · zero until first completion
00 00 00 00 00 00 00 00 00 08 00 00 00 00 00 00 00 00
Offsets are decimal, values are little-endian hex. user_data at SQE +32 returns at CQE +0.
Single issuer; 4-entry SQ/CQ, default layouts and SQ index array. Independent synthetic sockets/buffers; each operation completes with 512 bytes. The application waits for CQ space before dispatch; overflow, cancellation and SQPOLL are outside this model.
Notice that preparing an SQE alone does nothing. The application writes the description and the corresponding SQ array index, then publishes a larger SQ tail. With ordinary setup, io_uring_enter tells the kernel how many published entries to submit; it can also wait for completions. The kernel may complete an operation inline, wait for readiness and retry it, or use asynchronous workers where needed. “Asynchronous API” does not mean “every operation gets a dedicated background thread.”
For our request, the application submits a read with a buffer address and connection ID in user_data. On completion it reads res. A positive value is the byte count, zero can mean EOF, and a negative value is directly -errno. A short read still needs parser state. Once the header is complete, the application constructs the reply and submits a write. Only its write completion releases the output buffer for reuse, subject to the specific operation’s completion contract.
Ordinary read and write buffers remain valid until their operations complete. Closing a handle or dropping a language object is not a substitute for managing that lifetime. A cancellation request is itself asynchronous, and can race with successful completion. Use stable operation IDs to reconcile results. If you recycle a connection record too early, a late CQE can otherwise appear to belong to a new connection using the same descriptor number.
The bytes have an ABI
The table gives the fields used here, from Linux 7.3.4’s UAPI header. Unions give these byte positions different meanings for other opcodes. Zeroing an SQE before filling its selected fields avoids accidentally carrying flags from its previous use. Socket READ/WRITE use offset zero here; the descriptor is a socket rather than a seekable file.
| Record | Byte offset | Width | Used as |
|---|---|---|---|
| SQE | 0 / 1 / 2 | 1 / 1 / 2 | opcode / flags / ioprio |
| SQE | 4 / 8 / 16 | 4 / 8 / 8 | signed fd / offset union / buffer address |
| SQE | 24 / 28 / 32 | 4 / 4 / 8 | length / operation flags / user_data |
| SQE | 40 / 42 / 44 / 48 | 2 / 2 / 4 / 16 | buffer index union / personality / operation union / final unions |
| CQE | 0 / 8 / 12 | 8 / 4 / 4 | user_data / signed result / flags |
Two queues, four ordering edges
Each ring has a producer and consumer. The application produces SQ entries; the kernel consumes them. The kernel produces CQ entries; the application consumes them. The producer must finish writing an entry before publishing a tail that makes it visible. The consumer must finish reading an entry before publishing a head that makes its slot reusable. Atomic index access without ordering the record’s bytes is insufficient.
write SQE + array → release SQ tailkernel acquire SQ tail → read SQEconsume SQE → release SQ headapplication acquire SQ head → reuse SQ slotwrite CQE → release CQ tailapplication acquire CQ tail → read CQEread CQE → release CQ headkernel observes CQ head → reuse CQ slotThe barrier contract is spelled out at the start of io_uring/io_uring.c. In particular, acquire SQ head before overwriting reusable entries, release SQ tail after filling them, acquire CQ tail before reading completions, and release CQ head after consuming them. The kernel implements the matching ordering. The atomics dive’s two ownership handoffs explain why publication and reuse both need an ordering edge.
Indices are unsigned counters, not pointers. A ring with capacity N uses a mask N − 1 when N is a power of two. In real rings, 32-bit head and tail wrap, so occupancy uses unsigned modular subtraction with the capacity invariant maintained. The widget caps itself at 10,000 operations, well before counter wrap, but wraps physical slots every four entries. A full SQ cannot overwrite unconsumed work.
A full CQ needs a policy as well. The widget makes the application stop dispatching until it has CQ space, a deliberate flow-control policy. Real kernels have completion overflow handling, feature flags and counters; old or unsupported paths can lose completions under overflow. Applications size queues, reap promptly and monitor overflow rather than assuming that an arbitrary completion stream fits. Multishot operations make CQ demand particularly different from SQ occupancy.
SQ capacity and operation lifetime are different resources. Once the kernel has consumed an SQE, its SQ slot may be reusable even though the read described by it is still waiting for a packet. The original data buffer must remain alive. Returning SQ head therefore frees a description slot, while a later CQE ends the associated operation under its completion contract. Keeping separate pools for descriptions, operations and data buffers makes that distinction visible in an implementation.
Likewise, reaping a CQE returns CQ storage, not a connection. A read completion can mean that the handler needs another read to finish its header. A write completion can leave more bytes to send. The connection becomes reusable only when its protocol state and all outstanding operations permit it. This is why an operation ID and a connection ID often need a generation counter or another lifetime discipline in a long-running server.
What the faster modes change
SQPOLL starts a kernel submission polling thread. While it is awake, publishing a tail can be enough to start work without a submission syscall. When it sleeps, the application must wake it. The application publishes the tail, executes the required full memory barrier, then checks IORING_SQ_NEED_WAKEUP and calls enter with IORING_ENTER_SQ_WAKEUP when needed. Checking the flag before publication can race with the poller going to sleep.
That removes some boundary crossings by spending kernel CPU on polling. It does not make I/O or cache coherence free. Registered files amortise reference lookup through an explicit file table; registered buffers amortise pinning and mapping work for supported operations. Buffer registration does not by itself eliminate data copies. The costs move into setup, pinned resources, lifetime rules and eventual teardown.
Multishot accept can produce several accepted connections from one submission. Multishot receive can produce several receive completions using provided buffers, under its own flag and buffer-selection requirements. A CQE’s IORING_CQE_F_MORE flag indicates that the operation remains active. When MORE is absent, the application must account for termination and resubmit if appropriate. One SQE therefore need not imply one CQE.
Linked SQEs express dependencies inside a submitted chain. IOSQE_IO_LINK orders a successor after its predecessor and can cancel successors when the link breaks. It cannot parse an HTTP request or compute a response length from a future read result. In our server, user code must inspect the received header before deciding the next write. Ordinary independent submissions can complete out of order; user_data tells you which operation finished.
Shared memory replaces per-operation argument passing with an ownership protocol. The protocol, including buffer lifetimes and memory ordering, is part of the API.
Count the work you actually asked for
Return to 10,000 connections. Concurrency counts how many connections exist together; it does not tell you how many requests are ready together. A thousand idle sockets offer nothing to batch. Ten thousand simultaneously ready requests may let you amortise many crossings, provided the application has enough work and completion capacity and its latency budget tolerates batching.
For a transparent baseline, suppose all sockets are already accepted and registered, each complete request fits in one successful read, and each response fits in one successful write. Let R be 10,000 requests and B the batch capacity. There are W = ⌈R/B⌉ batches. Ignore EAGAIN, partial transfers, signals, setup and close for this accounting exercise. The thread-per-connection design uses 2R data syscalls. The epoll loop uses the same 2R plus W waits.
Our io_uring model submits one wave of reads per batch, handles their completions, then submits one wave of dependent writes. With one enter per wave and completions available before another wait is needed, it uses 2W enter calls. That is an achievable accounting pattern, not a lower bound or a universal benchmark. Real arrivals can require additional enter calls to wait or collect partial work. SQPOLL can reduce submission calls further while using another core’s time.
Syscalls per request · common scale 0 to 3
10,000 requests: 20,000 / 20,313 / 626 syscalls.
Model: accepted sockets; one successful read and write; one epoll wait per batch; two io_uring submission waves. Setup, retries and extra waits are excluded.
Cthreads = 2R · Cepoll = 2R + ⌈R/B⌉ · Curing = 2⌈R/B⌉
At B = 32, the model gives 20,000, 20,313 and 626 syscalls. epoll’s slightly larger count does not make it worse at handling 10,000 idle connections: it trades additional readiness calls for far fewer blocked threads. io_uring’s smaller count does not prove lower latency: parsing, copies, completion handling, memory ownership and queueing still take time. Measure the workload whose bottleneck you intend to remove.
Batching also introduces a latency decision. If only one request is ready, waiting for 31 more before submitting it reduces calls at the expense of that request’s response time. A practical loop submits available work and caps the batch, rather than treating the cap as a minimum fill requirement. Under high load that often creates large batches naturally; under light load it preserves prompt service. The slider changes a capacity assumption, while arrival timing decides how much of it the real server can use.
Run the three paths
The source below is a bounded loopback HTTP demonstrator with three modes. It accepts all requested clients before starting its data path, keeping the accepted-socket premise explicit. It handles fragmented header terminators up to 4 KiB, partial reads and writes, and closes after one fixed response. It binds an ephemeral loopback port, caps the demonstration at 256 clients, and has a ten-second process deadline. The test used 16 clients rather than consuming the host’s resources with 10,000 threads.
The same file contains the thread worker, a level-triggered epoll loop, and a raw io_uring implementation. The ring path maps the kernel-returned offsets and uses GCC acquire/release builtins on shared indices. It uses no liburing. The SQ submission function is worth reading before the complete source: payload bytes and array entry are ready before tail publication.
static void queue(struct ring *r, struct connection *c, unsigned id) {
unsigned tail = *r->st;
if (tail - acquire(r->sh) > *r->sm) {
fprintf(stderr, "SQ capacity exceeded\n"); exit(1);
}
unsigned slot = tail & *r->sm;
struct io_uring_sqe *e = (struct io_uring_sqe *)r->sqes + slot;
memset(e, 0, sizeof(*e));
e->opcode = c->writing ? IORING_OP_WRITE : IORING_OP_READ;
e->fd = c->fd;
e->addr = (uintptr_t)(c->writing ? reply + c->sent : c->buf + c->used);
e->len = (unsigned)(c->writing ? sizeof(reply) - 1 - c->sent : 4096 - c->used);
e->user_data = id;
r->array[slot] = slot;
release(r->st, tail + 1);
}gcc -O2 -Wall -Wextra -Werror bench.c -o bench
taskset -c 0 ./bench
gcc -O2 -Wall -Wextra -Werror -pthread server.c -o server
python3 client.py
# Save server.c and client.py in the same scratch directory.
# The driver launches all three modes and supplies 16 clients per run.Complete compilable server.c · all three modes
#define _GNU_SOURCE
#include <arpa/inet.h>
#include <errno.h>
#include <fcntl.h>
#include <linux/io_uring.h>
#include <pthread.h>
#include <signal.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/epoll.h>
#include <sys/mman.h>
#include <sys/socket.h>
#include <sys/syscall.h>
#include <unistd.h>
static const char reply[] =
"HTTP/1.1 200 OK\r\nContent-Length: 3\r\nConnection: close\r\n\r\nok\n";
struct connection {
int fd, writing, done;
size_t used, sent;
char buf[4097];
};
static void die(const char *s) {
perror(s);
exit(1);
}
static int headers(struct connection *c) {
c->buf[c->used] = 0;
return strstr(c->buf, "\r\n\r\n") != NULL;
}
static void finish(struct connection *c) {
close(c->fd);
c->done = 1;
}
static void advance(struct connection *c) {
while (!c->writing) {
ssize_t n = read(c->fd, c->buf + c->used, 4096 - c->used);
if (n < 0 && errno == EINTR) continue;
if (n < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) return;
if (n <= 0) {
finish(c);
return;
}
c->used += (size_t)n;
if (headers(c)) c->writing = 1;
else if (c->used == 4096) {
finish(c);
return;
}
}
while (c->sent < sizeof(reply) - 1) {
ssize_t n = write(c->fd, reply + c->sent, sizeof(reply) - 1 - c->sent);
if (n < 0 && errno == EINTR) continue;
if (n < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) return;
if (n <= 0) {
finish(c);
return;
}
c->sent += (size_t)n;
}
finish(c);
}
static void *worker(void *p) {
advance(p);
return NULL;
}
struct ring {
int fd;
void *sqmap, *cqmap, *sqes;
size_t sqsize, cqsize, esize;
unsigned *sh, *st, *sm, *array, *ch, *ct, *cm;
struct io_uring_cqe *cqes;
};
static unsigned acquire(unsigned *p) {
return __atomic_load_n(p, __ATOMIC_ACQUIRE);
}
static void release(unsigned *p, unsigned n) {
__atomic_store_n(p, n, __ATOMIC_RELEASE);
}
static void *map(int fd, size_t size, off_t offset) {
void *p = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, offset);
if (p == MAP_FAILED) die("mmap");
return p;
}
static struct ring setup(unsigned entries) {
struct io_uring_params p = {0};
int fd = (int)syscall(SYS_io_uring_setup, entries, &p);
if (fd < 0) die("io_uring_setup");
struct ring r = {.fd = fd};
r.sqsize = p.sq_off.array + p.sq_entries * sizeof(unsigned);
r.cqsize = p.cq_off.cqes + p.cq_entries * sizeof(struct io_uring_cqe);
r.esize = p.sq_entries * sizeof(struct io_uring_sqe);
r.sqmap = map(fd, r.sqsize, IORING_OFF_SQ_RING);
r.cqmap = map(fd, r.cqsize, IORING_OFF_CQ_RING);
r.sqes = map(fd, r.esize, IORING_OFF_SQES);
char *s = r.sqmap, *c = r.cqmap;
r.sh = (unsigned *)(s + p.sq_off.head);
r.st = (unsigned *)(s + p.sq_off.tail);
r.sm = (unsigned *)(s + p.sq_off.ring_mask);
r.array = (unsigned *)(s + p.sq_off.array);
r.ch = (unsigned *)(c + p.cq_off.head);
r.ct = (unsigned *)(c + p.cq_off.tail);
r.cm = (unsigned *)(c + p.cq_off.ring_mask);
r.cqes = (struct io_uring_cqe *)(c + p.cq_off.cqes);
return r;
}
static void queue(struct ring *r, struct connection *c, unsigned id) {
unsigned tail = *r->st;
if (tail - acquire(r->sh) > *r->sm) {
fprintf(stderr, "SQ capacity exceeded\n"); exit(1);
}
unsigned slot = tail & *r->sm;
struct io_uring_sqe *e = (struct io_uring_sqe *)r->sqes + slot;
memset(e, 0, sizeof(*e));
e->opcode = c->writing ? IORING_OP_WRITE : IORING_OP_READ;
e->fd = c->fd;
e->addr = (uintptr_t)(c->writing ? reply + c->sent : c->buf + c->used);
e->len = (unsigned)(c->writing ? sizeof(reply) - 1 - c->sent : 4096 - c->used);
e->user_data = id;
r->array[slot] = slot;
release(r->st, tail + 1);
}
static void completions(struct connection *cs, unsigned count) {
struct ring r = setup(count);
for (unsigned i = 0; i < count; i++) queue(&r, &cs[i], i);
unsigned left = count;
while (left) {
unsigned pending = *r.st - acquire(r.sh);
int n = (int)syscall(SYS_io_uring_enter, r.fd, pending, 1,
IORING_ENTER_GETEVENTS, NULL, 0);
if (n < 0) {
if (errno == EINTR) continue;
die("io_uring_enter");
}
unsigned head = *r.ch, tail = acquire(r.ct);
while (head != tail) {
struct io_uring_cqe event = r.cqes[head++ & *r.cm];
struct connection *c = &cs[event.user_data];
int result = event.res;
if (result == -EINTR || result == -EAGAIN) queue(&r, c, (unsigned)event.user_data);
else if (result <= 0) {
finish(c);
left--;
}
else {
if (c->writing) c->sent += (size_t)result;
else {
c->used += (size_t)result;
c->writing = headers(c);
}
if ((c->writing && c->sent == sizeof(reply) - 1) || (!c->writing && c->used == 4096)) {
finish(c);
left--;
} else queue(&r, c, (unsigned)event.user_data);
}
}
release(r.ch, head);
}
munmap(r.sqmap, r.sqsize);
munmap(r.cqmap, r.cqsize);
munmap(r.sqes, r.esize);
close(r.fd);
}
int main(int argc, char **argv) {
if (argc != 3 || (strcmp(argv[1], "threads") && strcmp(argv[1], "epoll") && strcmp(argv[1], "uring"))) {
fprintf(stderr, "usage: server threads|epoll|uring connections(1..256)\n"); return 1;
}
char *end;
long parsed = strtol(argv[2], &end, 10);
if (*end || parsed < 1 || parsed > 256) return 1;
unsigned count = (unsigned)parsed;
signal(SIGPIPE, SIG_IGN);
alarm(10);
int listener = socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0);
if (listener < 0) die("socket");
int yes = 1;
setsockopt(listener, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes));
struct sockaddr_in a = {.sin_family = AF_INET, .sin_port = 0,
.sin_addr.s_addr = htonl(INADDR_LOOPBACK)};
if (bind(listener, (void *)&a, sizeof(a)) || listen(listener, 256)) die("listen");
socklen_t alen = sizeof(a);
if (getsockname(listener, (void *)&a, &alen)) die("getsockname");
printf("%u\n", ntohs(a.sin_port));
fflush(stdout);
struct connection *cs = calloc(count, sizeof(*cs));
if (!cs) die("calloc");
for (unsigned i = 0; i < count; i++) {
cs[i].fd = accept4(listener, NULL, NULL, SOCK_CLOEXEC);
if (cs[i].fd < 0) die("accept4");
}
close(listener);
if (!strcmp(argv[1], "threads")) {
pthread_t tids[256];
for (unsigned i = 0; i < count; i++) {
int error = pthread_create(&tids[i], NULL, worker, &cs[i]);
if (error) {
errno = error;
die("pthread_create");
}
}
for (unsigned i = 0; i < count; i++) pthread_join(tids[i], NULL);
} else if (!strcmp(argv[1], "uring")) completions(cs, count);
else {
int ep = epoll_create1(EPOLL_CLOEXEC);
if (ep < 0) die("epoll_create1");
for (unsigned i = 0; i < count; i++) {
if (fcntl(cs[i].fd, F_SETFL, O_NONBLOCK)) die("fcntl");
struct epoll_event e = {.events = EPOLLIN, .data.u32 = i};
if (epoll_ctl(ep, EPOLL_CTL_ADD, cs[i].fd, &e)) die("epoll_ctl ADD");
}
unsigned left = count;
while (left) {
struct epoll_event events[256];
int n = epoll_wait(ep, events, 256, 5000);
if (n < 0 && errno == EINTR) continue;
if (n <= 0) die("epoll_wait");
for (int j = 0; j < n; j++) {
struct connection *c = &cs[events[j].data.u32];
advance(c);
if (c->done) left--;
else {
struct epoll_event e = {.events = c->writing ? EPOLLOUT : EPOLLIN,
.data.u32 = events[j].data.u32};
if (epoll_ctl(ep, EPOLL_CTL_MOD, c->fd, &e)) die("epoll_ctl MOD");
}
}
}
close(ep);
}
free(cs);
return 0;
}Complete client.py · reproduce all nine capture runs
import pathlib
import socket
import subprocess
import time
root = pathlib.Path(__file__).resolve().parent
request = b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n"
expected = (
b"HTTP/1.1 200 OK\r\nContent-Length: 3\r\n"
b"Connection: close\r\n\r\nok\n"
)
trace_calls = (
"trace=read,write,accept4,epoll_wait,epoll_ctl,"
"io_uring_setup,io_uring_enter,clone,clone3"
)
for mode in ["threads", "epoll", "uring"]:
for kind in ["trace", "summary", "time"]:
command = [str(root / "server"), mode, "16"]
if kind == "trace":
command = [
"strace", "-f", "-o", str(root / (mode + ".trace")),
"-e", trace_calls,
] + command
elif kind == "summary":
command = [
"strace", "-f", "-c", "-o", str(root / (mode + ".summary")),
] + command
else:
command = [
"/usr/bin/time", "-v", "-o", str(root / (mode + ".time")),
] + command
process = subprocess.Popen(
command,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
port = int(process.stdout.readline())
clients = [
socket.create_connection(("127.0.0.1", port), timeout=3)
for _ in range(16)
]
# All sockets exist before any handler finishes; split every header.
for client in clients:
client.sendall(request[:5])
time.sleep(0.01)
for client in clients:
client.sendall(request[5:])
for client in clients:
response = b""
while True:
data = client.recv(4096)
if not data:
break
response += data
assert response == expected, (mode, response)
client.close()
output, error = process.communicate(timeout=12)
assert process.returncode == 0, (mode, error)
print(mode, kind, "16/16 responses; fragmented headers handled")Save server.c and client.py together in a scratch directory, compile the server, then run the driver. It launches each child command below, connects all 16 clients, splits their headers with a 10 ms pause, and verifies every complete response. Trace, summary and timing files appear beside the driver. Counts can vary with scheduling.
# These are the child commands client.py runs, with its clients active.
for mode in threads epoll uring; do
strace -f -o "$mode.trace" -e trace=read,write,accept4,epoll_wait,epoll_ctl,io_uring_setup,io_uring_enter,clone,clone3 ./server "$mode" 16
strace -f -c -o "$mode.summary" ./server "$mode" 16
/usr/bin/time -v -o "$mode.time" ./server "$mode" 16
doneComplete compilable bench.c · the boundary measurement
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/syscall.h>
#include <time.h>
#include <unistd.h>
__attribute__((noinline)) long local_call(long x) {
__asm__ volatile("" : "+r"(x));
return x;
}
long raw_getpid(void) {
long result;
__asm__ volatile("syscall" : "=a"(result) : "a"(SYS_getpid)
: "rcx", "r11", "memory");
return result;
}
static uint64_t now(void) {
struct timespec t;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &t)) exit(1);
return (uint64_t)t.tv_sec * 1000000000 + t.tv_nsec;
}
int main(void) {
const int n = 200000;
int fd = open("/dev/zero", O_RDONLY);
if (fd < 0) {
perror("open");
return 1;
}
char byte;
volatile long sink = 0;
for (int mode = 0; mode < 4; mode++) {
uint64_t start = now();
for (int i = 0; i < n; i++) {
if (mode == 0) sink = local_call(i);
if (mode == 1) sink = getpid();
if (mode == 2) sink = raw_getpid();
if (mode == 3 && read(fd, &byte, 1) != 1) return 1;
}
printf("%s %.2f ns/call (%d iterations)\n",
(const char *[]) {"function", "glibc getpid", "raw getpid", "read /dev/zero"}[mode],
(double)(now() - start) / n, n);
}
close(fd);
return sink == -1;
}| Observed for 16 clients | Threads | epoll | io_uring |
|---|---|---|---|
read | 33 | 49 | 1 |
write | 17 | 17 | 1 |
epoll_wait | 0 | 2 | 0 |
epoll_ctl | 0 | 32 | 0 |
io_uring_enter | 0 | 0 | 4 |
clone3 | 16 | 0 | 0 |
total | 303 | 196 | 92 |
| Voluntary switches | 37 | 11 | 16 |
| Involuntary switches | 9 | 2 | 0 |
Syscalls: separate strace -f -c runs, including startup and close. Switches: /usr/bin/time -v without strace, including process threads. Each run splits every request header with a 10 ms client pause.
The strace counts include the loader’s one read and the startup write that prints the listening port. The thread run made 33 reads: one loader read plus two request fragments per client. The epoll run made 49 reads: those same reads plus 16 EAGAIN results while headers were incomplete. Its 32 control calls are 16 initial registrations and 16 modifications. The ring run shows one ordinary read and one write for startup, while request I/O appears as submitted work behind four io_uring_enter calls.
The aggregate totals include much more than the data path: 16 accepts, closes, mappings, thread setup and dynamic linking. The summaries were produced with strace -f -c; separate strace -fruns recorded individual calls. Context-switch rows come from separate untraced/usr/bin/time -v runs. They include setup, the client pause and nondeterministic host scheduling. They are observations, not constants to multiply by 625 for a 10,000-client prediction. perf stat -e cycles,instructions,context-switches was denied under the host’s perf_event_paranoid=4 setting.
1807764 read(9, "GET /", 4096) = 5
1807764 read(9 <unfinished ...>
1807764 <... read resumed>, " HTTP/1.1\r\nHost: localhost\r\n\r\n", 4091) = 30
1807764 write(9, "HTTP/1.1 200 OK\r\nContent-Length:"..., 60 <unfinished ...>
1807764 <... write resumed>) = 60
1807764 +++ exited with 0 +++Where Go fits
Go’s net.Conn.Read gives a goroutine blocking control flow over a nonblocking socket. If the syscall returns EAGAIN, the runtime can park that goroutine and retain the continuation on its growable stack. Linux epoll later reports readiness, and the runtime makes the goroutine runnable so it can retry the read. The OS thread can execute other goroutines while this one waits. The Go netpoller dive follows that exact path.
This combines the readability of a handler with the waiting behaviour of an event loop. It does not mean every syscall can park only a goroutine; runtime integration and the resource type matter. It also does not make 10,000 live connections free. Each still needs a socket, application state and buffers. Choose a model by the work you can simplify and the bottleneck you can measure, then verify the tail latency and resource use at realistic load.
Count syscalls to understand crossings, context switches to understand task scheduling, and elapsed time to understand the experience. None is a substitute for the others.
The instruction and the entry trampoline
The interface costs have been visible throughout, but what actually happens at the instruction? On the classic x86-64 path, syscall saves the next user instruction address in RCX and user flags in R11, applies a configured flags mask, establishes kernel privilege and transfers control to the address programmed into IA32_LSTAR. It does not validate a Linux descriptor, copy a request or select a scheduler task. Those are kernel software responsibilities.
The instruction also does not automatically load a Linux kernel stack pointer from a per-thread structure. Entry assembly must safely preserve the user stack pointer and establish the kernel stack. On the traditional Linux path, swapgs makes kernel per-CPU data accessible through the appropriate GS base. The entry sequence then constructs saved register state and proceeds into the C machinery. These details are architecture-specific; AArch64 exception entry uses different registers and vector machinery.
The raw stub below is real output from this host. GCC emits ENDBR64 under the distribution’s control-flow protection defaults, loads syscall number 39 into EAX, executes SYSCALL, then returns to its C caller. The benchmark’s raw path may inline this sequence inside its loop; the disassembled symbol shows the separately emitted function. Neither version includes libc’s errno conversion, which is unnecessary for a successful getpid.
00000000000014d0 <raw_getpid>:
14d0: f3 0f 1e fa endbr64
14d4: b8 27 00 00 00 mov $0x27,%eax
14d9: 0f 05 syscall
14db: c3 retlong raw_getpid(void) {
long result;
__asm__ volatile("syscall" : "=a"(result) : "a"(SYS_getpid)
: "rcx", "r11", "memory");
return result;
}entry_SYSCALL_64 · Linux 7.3.4 · selected source lines
swapgsSelect the kernel GS base for per-CPU access.movq %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)Preserve the user stack pointer in per-CPU scratch storage.SWITCH_TO_KERNEL_CR3 scratch_reg=%rspSwitch page tables when the configured isolation path requires it.movq PER_CPU_VAR(cpu_current_top_of_stack), %rspLoad this task’s kernel stack top.pushq %raxSave orig_ax after the SS, user RSP, flags, CS and RIP pushes.PUSH_AND_CLEAR_REGS rax=$-ENOSYSComplete pt_regs; initialise its saved result to -ENOSYS.movq %rsp, %rdiPass the pt_regs pointer as C argument one.movslq %eax, %rsiSign-extend EAX into the syscall-number argument.call do_syscall_64Run entry policy, dispatch and return work in C.
Instructions are quoted in source order; intervening frame pushes and mitigation macros are omitted. This is kernel assembly, not a callable user-space sample. Read the complete entry path.
The entry point is only the beginning
Linux’s entry_64.S surrounds the useful operation with state management and mitigation sequences. Entry work can include seccomp, tracing, auditing and signal-related handling, depending on configuration and process state. Return work checks whether the fast return is safe and may choose an IRET path instead of SYSRET. Measuring an empty syscall includes that machinery’s active path, not merely the decoder latency of one instruction.
The architectural instruction contract is documented in the Intel software developer manuals. Internally a processor can implement parts of the transition through microcoded sequences and model-specific control logic. The manuals do not give a portable fixed micro-op count or a universal cycle cost for Linux entry and exit. CPU generation, microcode revision, active mitigations and kernel code all change the result. A single “syscall microcode takes N cycles” label would collapse those independent costs.
Our host uses the classic path. Newer x86 processors and kernels can use FRED, Flexible Return and Event Delivery, which changes event-entry mechanics. Read the entry code selected by your actual configuration before treating the LSTAR diagram as the only possible x86 path. The syscall ABI is much more stable than its internal implementation.
What Meltdown changed
Page-table isolation, commonly KPTI, separates user-visible mappings from the kernel’s full mapping on affected systems. Entry switches to kernel page tables; return switches back. This adds instructions and can affect translation caching. PCID and related hardware features help retain useful translations across address-space changes. It is too strong to say that every syscall flushes the whole TLB, or that a privilege transition automatically flushes the data cache.
The point of KPTI is to reduce exposure to attacks such as Meltdown, not to accelerate calls. CPUs that report themselves unaffected may avoid that mitigation. The local host’s vulnerability report says Not affected for Meltdown. Its syscall measurement therefore cannot stand in for an older vulnerable machine with KPTI enabled. The memory dive’s TLB model explains what a lost translation costs without repeating that model here.
Speculative branch mitigations address a different set of problems. Retpolines change some indirect branch sequences; IBRS/eIBRS constrain prediction behaviour across relevant boundaries; IBPB can invalidate prediction state at selected security transitions. They are not three fixed taxes paid on every syscall. This host reports enhanced/automatic IBRS and conditional IBPB, along with other model-specific protections. Kernel configuration and CPU capabilities choose which sequences apply.
Ordinary cache effects remain even without a mandatory flush. Entering kernel code brings its instructions and data into the hierarchy, and later user code may compete with them. A task switch can add a different working set, and migration can make that working set colder. Shared rings reduce some argument and result copying, but their indices and entries still travel through coherent caches. The atomics dive shows why a shared index has a hardware cost even when its machine instruction is just a load or store.
The packet that makes the read runnable
There is one final actor outside the handler: the packet that makes read useful. For a physical interface, the NIC receives a frame and uses DMA to place data into host memory described by receive descriptors. A receive interrupt can notify a core, but high packet rates make interrupting once for every packet wasteful. Devices and drivers batch notifications and work.
Linux’s NAPI lets the driver schedule polling work after a notification. A poll processes up to its receive-packet budget and may reclaim any number of transmit completions. It commonly runs in software interrupt context, with other execution modes available. Under load, work can continue through deferred processing instead of recursively monopolising the original interrupt handler. Busy polling is another option, trading CPU time for reduced notification latency.
The packets then pass through the network stack. TCP validates sequence information and makes ordered stream bytes available to the socket. The arrival that matters to our handler is not simply “a frame reached the NIC.” It is that the socket’s receive condition changed enough to let an operation progress. Reordering, loss, receive thresholds and scheduling can separate those moments.
Realtek r8169 · Linux 7.3.4 · 16-byte RxDesc
Length: bits 13–0
Status / frame flags
- Driver gives NIC ownership · OWN = 1Map the buffer, store its DMA address, then use dma_wmb() before publishing OWN and buffer capacity.
- NIC gives driver ownership · OWN = 0DMA fills the buffer; the descriptor reports received length/status and clears OWN.
- CPU consumes in NAPI poll, then returns ownershiprtl8169_poll() calls rtl_rx(). Check OWN, use dma_rmb(), synchronise DMA for CPU access, copy bytes to an skb, and pass it to GRO. Synchronise for the device and rearm OWN = 1.
Source: r8169_main.c: RxDesc, rtl8169_mark_to_asic(), rtl_rx(), rtl8169_poll(). Other NICs use different descriptors, completion formats and buffer-recycling paths.
The socket’s data-ready callback bridges networking into waiting machinery. For the threaded server, a wakeup makes the blocked reader runnable. For epoll, the installed callback can place the watch on a ready list and wake an epoll waiter. For an io_uring request waiting on socket readiness, notification can drive a retry and eventual completion. The socket callback and wait infrastructure live in net/core/sock.c and the relevant polling implementation.
All three designs ultimately contend for CPU service, memory bandwidth and socket state. Pinning a worker near the receive queue that feeds it can improve locality, but balancing affinity also affects load distribution. Hardware receive-side scaling, software steering and application worker placement determine where packets and handlers run. Moving a connection’s work between cores can move cache lines even when the syscall count stays unchanged.
Now trace the reply outward. The handler’s write, or its io_uring write operation, copies or references data under the chosen API’s rules and queues it into the local send path. A positive result means local progress. It does not mean the remote application parsed the response, and it does not imply a TCP acknowledgement arrived. Backpressure appears when local buffering cannot accept more data; then the thread sleeps, the epoll handler retains its output state, or the asynchronous operation remains pending.
That is why completion must always be read with its object and operation in mind. A write CQE reports that write’s result. A TCP ACK reports peer transport acknowledgement. An HTTP response observed by a client reports application-visible delivery. They happen at different points and answer different questions. Treating all three as “the request is done” produces misleading latency measurements.
The 16-client tests ran over loopback, so they are evidence about the local API paths and the sample’s handling of split headers. They do not measure NIC interrupt moderation, DMA, physical network round trips or a 10,000-client production load. A production experiment would hold request size, connection lifetime and offered load constant, then measure throughput, tail latency, resident memory, CPU consumption and queueing together. Syscall totals would explain part of that result rather than decide it in advance.
Our one request has kept its identity all the way down. At the top it was a blocking-looking read in a handler. At the boundary it was a number and registers. In the kernel it was a file object and a TCP receive condition. While waiting it was either a sleeping thread, an epoll watch plus application state, or an in-flight io_uring operation with stable buffers. At the hardware edge it became bytes arriving through device and cache machinery. Each design changes where the waiting and bookkeeping live; all must eventually move the same request bytes and produce the same reply.
The fastest waiting strategy preserves useful work and avoids unnecessary work. The packet, the parser and the lifetime rules still decide when a request can actually finish.
End of System calls · 8 levels, Idea to Silicon
Where to next
Next in Systems and hardware · 06 of 06
How a GPU runs a kernel, from threadIdx to DRAM
Follow a CUDA thread into a warp and onto an SM. Make branches diverge, scatter a memory request across sectors, and fix a transpose one bank conflict at a time.
Practical notes
Find what a process is waiting on with strace
Trace the blocked syscall, follow its file descriptor and identify the thread or peer that must wake it. Use syscall duration rather than summary CPU time to measure waiting.