MCREngine: A scalable checkpointing system using data-aware aggregation and compression

Abstract

Abstract-High performance computing (HPC) systems use checkpoint-restart to tolerate failures. Typically, applications store their states in checkpoints on a parallel file system (PFS). As applications scale up, checkpoint-restart incurs high overheads due to contention for PFS resources. The high overheads force large-scale applications to reduce checkpoint frequency, which means more compute time is lost in the event of failure.We alleviate this problem through a scalable checkpointrestart system, MCRENGINE. MCRENGINE aggregates checkpoints from multiple application processes with knowledge of the data semantics available through widely-used I/O libraries, e.g., HDF5 and netCDF, and compresses them. Our novel scheme improves compressibility of checkpoints up to 115% over simple concatenation and compression. Our evaluation with large-scale application checkpoints show that MCRENGINE reduces checkpointing overhead by up to 87% and restart overhead by up to 62% over a baseline with no aggregation or compression.

Resilience Design Patterns: A Structured Approach to Resilience at Extreme Scale (V.2.0)

“…Incremental checkpointing dynamically identifies the changed blocks of memory since the last checkpoint through a hash function [12] in order to limit the amount of state required to be captured per checkpoint. Data aggregation and compression also help reduce the bandwidth requirements when committing the checkpoint to disk [104]. Plank et al eliminate the overhead of writing checkpoints to disk altogether with a diskless in-memory checkpointing approach [150].…”

Checkpointing Strategies for Shared High-Performance Computing Platforms

“…Finally, compression-based techniques use standard compression algorithms to reduce checkpoint volumes [29] and can be used at the compiler-level [30] or in-memory [31]. Related, Plank et al proposed differential compression to reduce checkpoint sizes for incremental checkpoints [32] and Tanzima et al show that similarities amongst checkpoint data from different processes can be exploited to compress and reduce checkpoint data volumes [33]. Finally, Sasaki et al propose a lossy compression method based on wavelet transform and vector quantization to the checkpoints of a production climate application [34], while Ni et al [35] study the trade-offs between the loss of precision, compression ratio, and application correctness due to lossy compression.…”

Optimal Cooperative Checkpointing for Shared High-Performance Computing Platforms

“…Finally, compression-based techniques use standard compression algorithms to reduce checkpoint volumes [27] and can be used at the compiler-level [28] or in-memory [29]. Related, Plank et al proposed differential compression to reduce checkpoint sizes for incremental checkpoints [30] and Tanzima et al show that similarities amongst checkpoint data from different processes can be exploited to compress and reduce checkpoint data volumes [31]. Finally, Sasaki et al propose a lossy compression method based on wavelet transform and vector quantization to the checkpoints of a production climate application [32], while Ni et al [33] study the trade-offs between the loss of precision, compression ratio, and application correctness due to lossy compression.…”