Transparent checkpoint-restart over infiniband

Abstract

InfiniBand is widely used for low-latency, high-throughput cluster computing. Saving the state of the InfiniBand network as part of distributed checkpointing has been a long-standing challenge for researchers. Because of a lack of a solution, typical MPI implementations have included custom checkpoint-restart services that "tear down" the network, checkpoint each node as if the node were a standalone computer, and then re-connect the network again. We present the first example of transparent, system-initiated checkpoint-restart that directly supports InfiniBand. The new approach is independent of any particular Linux kernel, thus simplifying the current practice of using a kernel-based module, such as BLCR. This direct approach results in checkpoints that are found to be faster than with the use of a checkpoint-restart service. The generality of this approach is shown not only by checkpointing an MPI computation, but also a native UPC computation (Berkeley Unified Parallel C), which does not use MPI. Scalability is shown by checkpointing 2,048 MPI processes across 128 nodes (with 16 cores per node). In addition, a cost-effective debugging approach is also enabled, in which a checkpoint image from an InfiniBand-based production cluster is copied to a local Ethernet-based cluster, where it can be restarted and an interactive debugger can be attached to it. This work is based on a plugin that extends the DMTCP (Distributed MultiThreaded CheckPointing) checkpoint-restart package.

System-Level Scalable Checkpoint-Restart for Petascale Computing

“…In testing on the LU.E benchmark, we saw runtime overhead rise to 9% for 4K CPU cores compared to the 1.7% runtime overhead at 2K cores reported by [13]. This was due to the non-scalable tracing of send/receive requests required by the InfiniBand checkpointing code to shadow hardware state, since InfiniBand devices don't provide a way to "peek" at the current state.…”

“…At 32,752 CPU cores, the tests use one-third of the compute nodes of Stampede. This is sixteen times as many cores as the largest previous transparent checkpoint of which we are aware [13]. The Stampede supercomputer is rated at 5.2 PFlops (RMAX:sustained) or 8.5 PFlops (RPEAK:peak).…”

“…This approach does not scale, since BLCR does not support SysV shared memory objects [12]. Most modern MPI implementations require such shared memory for efficient communication among MPI processes on the same node to avoid the delay in going through kernel system calls.Moreover, an important work on transparent, system-level checkpointing is [13], which supported only InfiniBand RC (reliable connection) mode. While that result sufficed for earlier MPI implementations, modern MPI implementations require InfiniBand UD for optimal performance when running with more than about 64 processes.…”

Checkpoint/restart approaches for a thread-based MPI runtime

“…For example, memory pinning register segments in the device (to allow address translation and to retrieve authentication tokens) is not checkpointable. In order to circumvent this issue, DMTCP provided a plugin completely wrapping the libverbs (low-level Infiniband programming interface) in order to track and preserve a shadow state of all the operations taking place on the card [11]. This approach enabled transparent checkpointing of Infiniband networks, but not without some drawbacks.…”

Transparent checkpointing over RDMA-based networks

“…As of this writing, Stampede is the #10 supercomputer on the Top500 list [78]. In all cases, each computer node was running This is sixteen times as many cores as the largest previous transparent checkpoint of which we are aware [15]. The Stampede supercomputer is rated at 5.2 PFlops (RMAX:sustained) or 8.5 PFlops (RPEAK:peak) [78].…”