CRAFT: A Library for Easier Application-Level Checkpoint/Restart and Automatic Fault Tolerance

Abstract

In order to efficiently use the future generations of supercomputers, fault tolerance and power consumption are two of the prime challenges anticipated by the High Performance Computing (HPC) community. Checkpoint/Restart (CR) has been and still is the most widely used technique to deal with hard failures. Application-level CR is the most effective CR technique in terms of overhead efficiency but it takes a lot of implementation effort. This work presents the implementation of our C++ based library CRAFT (Checkpoint-Restart and Automatic Fault Tolerance), which serves two purposes. First, it provides an extendable library that significantly eases the implementation of application-level checkpointing. The most basic and frequently used checkpoint data types are already part of CRAFT and can be directly used out of the box. The library can be easily extended to add more data types. As means of overhead reduction, the library offers a build-in asynchronous checkpointing mechanism and also supports the Scalable Checkpoint/Restart (SCR) library for node level checkpointing. Second, CRAFT provides an easier interface for User-Level Failure Mitigation (ULFM) based dynamic process recovery, which significantly reduces the complexity and effort of failure detection and communication recovery mechanism. By utilizing both functionalities together, applications can write application-level checkpoints and recover dynamically from process failures with very limited programming effort. This work presents the design and use of our library in detail. The associated overheads are thoroughly analyzed using several benchmarks.Thomas Zeiser holds a PhD in Computational Fluid Mechanics from the University of Erlangen-Nuremberg. He is now a senior research scientist in the HPC group of RRZE and is among many other things still interested in lattice Boltzmann methods.Georg Hager holds a PhD in Computational Physics from the University of Greifswald. He has been working with high performance systems since 1995, and is now a senior research scientist in the HPC group at RRZE. Recent research includes architecture-specific optimization for current microprocessors, performance modeling on processor and system levels, and the efficient use of hybrid parallel systems. His daily work encompasses all aspects of user support in HPC such as lectures, tutorials, training, code parallelization, profiling and optimization, and the assessment of novel computer architectures and tools.Gerhard Wellein holds a PhD in Solid State Physics from the University of Bayreuth and is a regular Professor at the Department for Computer Science at University of Erlangen. He heads the HPC group at RRZE and has more than 10 years of experience in teaching HPC techniques to students and scientists from Computational Science and Engineering. His research interests include solving large sparse eigenvalue problems, novel parallelization approaches, performance modeling, and architecture-specific optimization.

Benefits from using mixed precision computations in the ELPA-AEO and ESSEX-II eigensolver projects

“…At various places, measures for improving resilience have been included, based on verifying known properties of computed quantities and on checksums, combined with checkpoint-restart. To simplify incorporating the latter into numerical algorithms, the Checkpoint-Restart and Automatic Fault Tolerance (CRAFT) library has been developed [30]. Figure 3 illustrates its use within the BEAST framework.…”

Optimum Interval for Application-level Checkpoints

“…They are often divided into [7]: (i) system-level CR [37], (ii) library-level CR [38], and (iii) application-level CR (ALCR) [7]. ALCR [7], [21] is considered the most efficient, since it leaves the smallest memory footprint [7], [8], [22]; however it requires manual source code modifications for introducing checkpoints into the program.…”

“…Application Level Checkpoint and Restart (ALCR) is widely used to enhance the reliability of long-running programs [6]- [8] by periodically saving a copy or checkpoint of the current execution state of software. The most recent copy is then used to restart program execution in case of failure.…”

“…In [19], [20] the impact of asynchronous checkpointing strategies on the performance of distributed systems has been studied. Among the existing checkpointing strategies, ALCR [6], [21] uses a small memory footprint [7], [8], but requires significant expertise for the selection of source code locations in which checkpoints should be inserted. Yet existing ALCR tools and libraries facilitate the insertion of checkpoints in long-running loops since computational loops constitute a significant source of failure-related re-executions [22], [23].…”

Towards Ad Hoc Recovery for Soft Errors

“…The User Level Failure Mitigation (ULFM) [15] corresponds to the most recent effort for the inclusion of resilience capabilities in the MPI standard, enabling applications to detect and react to failures without stopping their execution. Several works have implement resilient applications using the ULFM features [16]- [22]. ULFM enables the deployment of different recovery strategies after repairing the communication environment when a failure hits the application, thus, avoiding the overheads of re-initializing the entire MPI application.…”