The Design and Implementation of Checkpoint/Restart Process Fault Tolerance for Open MPI

Abstract

To be able to fully exploit ever larger computing platforms, modern HPC applications and system software must be able to tolerate inevitable faults. Historically, MPI implementations that incorporated fault tolerance capabilities have been limited by lack of modularity, scalability and usability. This paper presents the design and implementation of an infrastructure to support checkpoint/restart fault tolerance in the Open MPI project. We identify the general capabilities required for distributed checkpoint/restart and realize these capabilities as extensible frameworks within Open MPI's modular component architecture. Our design features an abstract interface for providing and accessing fault tolerance services without sacrificing performance, robustness, or flexibility. Although our implementation includes support for some initial checkpoint/restart mechanisms, the framework is meant to be extensible and to encourage experimentation of alternative techniques within a production quality MPI implementation.

On the potential of significance-driven execution for energy-aware HPC

“…Methods for tolerating errors in hardware have been studied in the past [6,9]. This resulted in several solutions at different levels of the design stack, in both hardware and software.…”

The Architecture of the XtreemOS Grid Checkpointing Service

“…Some MPI libraries make use of system-level checkpointing in application-level checkpointing. Open MPI [13] and LAM/MPI [23] have chosen to implement generic checkpoint/restart mechanisms that can support multiple existing kernel checkpointers. Since this approach fits well with ours, these two libraries can be supported by our service, too.…”

“…The Open MPI Checkpoint/Restart framework [13] is very similar to XtreemGCP but limited to the MPI context. The framework architecture is based on a central coordinator, is able to address various process checkpointers.…”

Proactive process-level live migration and back migration in HPC environments

“…The implementation, while realized over LAM (Local Area Multicomputer)/MPI's C/R support [43] through Berkeley Labs C/R (BLCR) [15], is in its mechanisms applicable to any process-migration solution, e.g., the Open MPI FT mechanisms [24], [25]. BLCR is an open source, systemlevel C/R implementation integrated with LAM/MPI via a callback function.…”