Virtual-to-Physical address translation for an FPGA-based interconnect with host and GPU remote DMA capabilities

Abstract

We developed a custom FPGA-based Network Interface Controller named APEnet+ aimed at GPU accelerated clusters for High Performance Computing. The card exploits peer-to-peer capabilities (GPU-Direct RDMA) for latest NVIDIA GPGPU devices and the RDMA paradigm to perform fast direct communication between computing nodes, offloading the host CPU from network tasks execution. In this work we focus on the implementation of a Virtual to Physical address translation mechanism, using the FPGA embedded soft-processor. Address management is the most demanding task -we estimated up to 70% of the µC load -for the NIC receiving side, resulting being the main culprit for data bottleneck. To improve the performance of this task and hence improve data transfer over the network, we added a specialized hardware logic block acting as a Translation Lookaside Buffer. This block makes use of a peculiar Content Address Memory implementation designed for scalability and speed. We present detailed measurements to demonstrate the benefits coming from the introduction of such custom logic: a substantial address translation latency reduction (from a measured value of 1.9 µs to 124 ns) and a performance enhancement of both host-bound and GPU-bound data transfers (up to ∼ 60% of bandwidth increase) in given message size ranges.

ASIP acceleration for virtual-to-physical address translation on RDMA-enabled FPGA-based network interfaces

“…In [21] a number of CAM designs suitable for FPGA implementation are discussed. Our approach is a RAM-based design (in terms of resources used) but with a significant difference in the interpretation of the RAM ports semantics, as explained in [6].…”

“…Just like on x86_64 an associative cache called Translation Lookaside Buffer (TLB) assists the MMU in its duties, we followed suit and implemented a TLB for APEnet+ by means of a Content Addressable Memory (CAM) [6]. The CAM implementation allows the lowest possible latency for the TLB, which achieves actual saturation over the links.…”

“…In Section 4 the RDMA semantics for a network transfer is described. Section 5 contains specifics of the different page walking implementations; the Nios II firmware is in 5.1, the TLB is sketched in 5.2 -its in-depth examination can be found in [6] -and the ASIP is in 5.3; quantitative comparison between firmware and ASIP is in Section 6. Plans for complete integration of the page walking ASIP solution into APEnet+ and roadmap for its future development are presented in Section 7; conclusions are in Section 8.…”

Exploring Shared Virtual Memory for FPGA Accelerators with a Configurable IOMMU

“…In the HPC domain, SVM for letting FPGA accelerators themselves orchestrate data transfers from and to main memory has proven beneficial both for performance and programmability [3] [23]. While academic works focus on SVM solutions consisting of TLBs managed in software either running on the host [23] or on a dedicated soft processor core [24], the industry's approach is that of fullblown hardware for maximum performance [3] [4] [5]. To optimize off-chip bandwidth utilization between host and FPGA, such systems employ FPGA-side data caches, as well as transaction coalescing and reordering [25], which further increases design complexity and leads to considerable resource utilization.…”

NaNet3: The on-shore readout and slow-control board for the KM3NeT-Italia underwater neutrino telescope

“…The hardware follows the RDMA paradigm to manage the I/O towards both CPU and GPU -for this latter the protocol is more precisely the GPUDirect RDMA [5], -in this way avoiding the use of bounce buffers and minimizing the latency jitter of data transfers to and from application memory. The virtual-to-physical address translation is entrusted to a proprietary Translation Look-aside Buffer based on Content Addressable Memory [6]. In order to sustain the ∼ 320 MB/s aggregate bandwidth of the multi-channel system, incoming data are sent to the computing node by PCIe DMA write processes.…”