User Space Network Drivers

Abstract

The rise of user space packet processing frameworks like DPDK and netmap makes low-level code more accessible to developers and researchers. Previously, driver code was hidden in the kernel and rarely modified-or even looked atby developers working at higher layers. These barriers are gone nowadays, yet developers still treat user space drivers as black-boxes magically accelerating applications. We want to change this: every researcher building high-speed network applications should understand the intricacies of the underlying drivers, especially if they impact performance. We present ixy, a user space network driver designed for simplicity and educational purposes to show that fast packet IO is not black magic but careful engineering. ixy focuses on the bare essentials of user space packet processing: a packet forwarder including the whole NIC driver uses less than 1,000 lines of C code.This paper is partially written in tutorial style on the case study of our implementations of drivers for both the Intel 82599 family and for virtual VirtIO NICs. The former allows us to reason about driver and framework performance on a stripped-down implementation to assess individual optimizations in isolation. VirtIO support ensures that everyone can run it in a virtual machine.Our code is available as free and open source under the BSD license at https://github.com/emmericp/ixy.

The Case for Writing Network Drivers in High-Level Programming Languages

“…Moving to user space drivers also means moving to a microkernel architecture with an IPC mechanism between drivers and applications. User space drivers are more isolated from the rest of the system than kernel drivers and can even run without root privileges if IOMMU hardware is available [17,30].…”

“…We target Linux on x86 using the uio subsystem to map PCIe resources into user space programs. For details on implementing user space drivers, have a look at our original publication about the C driver [17].…”

“…All of our drivers implement the same basic architecture as our ixy user space driver [17] which serves as reference implementation that our high-level drivers are compared against. Ixy's architecture is inspired by DPDK, i.e., it uses poll mode drivers without interrupts, all APIs are based on batches of packets, and DMA buffers are managed in custom memory pools.…”

“…This interface enables us to use the IOMMU to isolate the device and run without root privileges and to use interrupts instead of polling under low load. Moreover, this interface offers a portable approach to allocating DMA memory in user space; the other implementations are specific to x86 and rely on an implementation detail in Linux (see [17]). [56] is a network function framework that allows users to build and compose network functions written in Rust.…”

“…All tests therefore use minimum-sized packets (up to 29.76 Mpps at 20 Gbit/s). We also benchmark the reference C driver ixy [17] as baseline performance measurement. The C driver proved to be as fast as older versions of DPDK but slower than modern versions of DPDK that offer a heavily vectorized transmit and receive path in its ixgbe driver [17].…”