Swapping to Remote Memory over InfiniBand: An Approach using a High Performance Network Block Device

Abstract

IntroductionTraditionally, operations with memory on other nodes (remote memory) in cluster environments interconnected with technologies like Gigabit Ethernet have been expensive with latencies several magnitudes slower than local memory accesses. Modern RDMA capable networks such as InfiniBand and Quadrics provide low latency of a few microseconds and high bandwidth of up to 10 Gbps. This has significantly reduced the latency gap between access to local memory and remote memory in modern clusters. Remote idle memory can be exploited to reduce the memory pressure on individual nodes. This is akin to adding an additional level in the memory hierarchy between local memory and the disk, with potentially dramatic performance improvements especially for memory intensive applications. In this paper, we take on the challenge to design a remote paging system for remote memory utilization in InfiniBand clusters. We present the design and implementation of a high performance networking block device (HPBD) over InfiniBandfabric, which serves as a swap device ofkernel Virtual Memory (VM) systemfor efficientpage transfer to/from remote memory servers. Our experiments show that using HPBD, quick sort performs only 1.45 times slower than local memory system, and up to 21 times faster than local disk. And our design is completely transparent to user applications. To the best ofour knowledge, it is thefirst work of a remote pager design using InfiniBandfor remote memory utilization.

MemX: Virtualization of Cluster-Wide Memory

“…In similar spirit, memcached [7] provides an API to access a large distributed in-memory key-value store. In non-virtualized settings, the use of memory from other machines to support large memory workloads has been explored earlier [19], [20], [21], [22], [23], [8], [24], primarily in 1990s. However, these systems did not address the comprehensive the design and performance considerations in using cluster-wide memory for virtual machine workloads.…”

Overcoming data locality: An in-memory runtime file system with symmetrical data distribution

“…Building a network block device attaching remote memory over InfiniBand had already been proposed in [6]. As a current example, the Dutch National Supercomputer, Cartesius [7], is equipped with Mellanox ConnectX-3 InfiniBand adapters that are able to achieve a unidirectional bandwidth of 56Gb/s.…”

A Cluster Computer Performance Predictor for Memory Scheduling

“…Shuang et al design a remote paging system for remote memory utilization in InfiniBand clusters [17]. In [18], the use of remote memory for virtual memory swapping in a cluster computer is described.…”