A scalable and extensible checkpointing scheme for massively parallel simulations

Abstract

Realistic simulations in engineering or in the materials sciences can consume enormous computing resources and thus require the use of massively parallel supercomputers. The probability of a failure increases both with the runtime and with the number of system components. For future exascale systems it is therefore considered critical that strategies are developed to make software resilient against failures. In this article, we present a scalable, distributed, diskless, and resilient checkpointing scheme that can create and recover snapshots of a partitioned simulation domain. We demonstrate the efficiency and scalability of the checkpoint strategy for simulations with up to 40 billion computational cells executing on more than 400 billion floating point values. A checkpoint creation is shown to require only a few seconds and the new checkpointing scheme scales almost perfectly up to more than 260 000 (2 18 ) processes. To recover from a diskless checkpoint during runtime, we realize the recovery algorithms using ULFM MPI. The checkpointing mechanism is fully integrated in a state-of-the-art high-performance multi-physics simulation framework. We demonstrate the efficiency and robustness of the method with a realistic phase-field simulation originating in the material sciences and with a lattice Boltzmann method implementation.

A Scalable and Modular Software Architecture for Finite Elements on Hierarchical Hybrid Grids

“…Such data structures could also for example store logistic data as required for handling special boundary conditions or metadata that is needed for load balancing. Similar approaches to decouple the simulation data from the coarse-grained topology have been presented in [7,27] and have been shown to provide an elegant interface for runtime load balancing [28], resilience, and data migration [29]. The macro-primitive graph data structure is therefore not restricted to finite-element based simulation techniques.…”

Exploring Parallel MPI Fault Tolerance Mechanisms for Phylogenetic Inference with RAxML-NG

“…Obersteiner et al (2017) extended a plasma simulation, Laguna et al (2016) a molecular dynamics simulation, and Engelmann and Geist (2003) a Fast Fourier Transformation that gracefully handle hardware faults. Kohl et al (2017) implemented a checkpoint-recovery system for a material science simulation. After a failure, the system initially assigns the work of the failed PEs to a single PE.…”

“…Researchers have already used ULFM for other scientific software (Ali et al, 2016;Engelmann and Geist, 2003;Kohl et al, 2017;Laguna et al, 2016;Obersteiner et al, 2017). ULFM reports failures by returning an error on at least one rank which participated in the failed communication.…”

Adaptive control in roll-forward recovery for extreme scale multigrid

“…In addition, when using checkpointing techniques, the additional time for each checkpoint delays the termination of the program and automatically yields a higher power consumption. Moreover, the memory necessary for checkpointing needs to be provided Kohl et al (2017) and must be kept in a reliable state, which may further increase energy consumption. Besides hard failures, which cause a physical loss of a computing entity, an increase of failures which are not immediately noticeable, is expected.…”

“…For the reconstruction of the static data, we refer to the in-memory checkpointing approaches and their performance analysis presented (e.g. Kohl et al, 2019).…”