A Fault Tolerance Protocol with Fast Fault Recovery

Abstract

Fault tolerance is an important issue for large machines with tens or hundreds of thousands of processors. Checkpoint-based methods, currently used on most machines, rollback all processors to previous checkpoints after a crash. This wastes a significant amount of computation as all processors have to redo all the computation from that checkpoint onwards. In addition, recovery time is bound by the time between the last checkpoint and the crash. Protocols based on message logging avoid the problem of rolling back all processors to their earlier state. However, the recovery time of existing message logging protocols is no smaller than the time between the last checkpoint and crash. We present a fault tolerance protocol, in this paper, that provides fast restarts by using the ideas of message logging and object-based processor virtualization. We evaluate our implementation of the protocol in the Charm++/Adaptive MPI runtime system. We show that our protocol provides fast restarts and, for many applications, has low fault-free overhead.

Assessing Energy Efficiency of Fault Tolerance Protocols for HPC Systems

“…If the system allows migratable tasks, then some of the tasks that live on the recovering node may be migrated to other nodes to be recovered in parallel. This scheme is an extension of messagelogging and is called parallel recovery [6]. It has the potential to reduce recovery time to a fraction of the normal rework time.…”

“…The second strategy is a particular version of message-logging [5] that requires messages to be stored, but avoids a global rollback in case of a failure. Finally, the third approach is called parallel recovery [6] and requires the system to allow tasks to migrate after a failure. This ability potentially reduces recovery time to a small fraction of re-execution time from checkpoint.…”

Proactive process-level live migration and back migration in HPC environments

“…The feasibility of proactive FT has been demonstrated at the job scheduling level [34] and in Adaptive MPI [8], [7], [9] using a combination of (a) object virtualization techniques to migrate tasks and (b) causal message logging [16] within the MPI runtime system of Charm++ applications. In contrast to Charm++, our solution is coarser grained as FT is provided at the process level, thereby encapsulating most of the process context, including open file descriptors, which are beyond the MPI runtime layer.…”

“…1 The feasibility of health monitoring at various levels has recently been demonstrated for temperature-aware monitoring, e.g., by using ACPI [1], and, more generically, by critical-event prediction [40]. Particularly in systems with thousands of processors, fault handling becomes imperative, yet approaches range from application-level and runtime-level to the level of OS schedulers [8], [7], [9], [34]. These and other approaches differ from our work in that we promote live migration combined with health monitoring.…”

Assessing HPC Failure Detectors for MPI Jobs

“…Our resulting implementation can easily be combined with reactive checkpoint/restart frameworks to trigger restarts after components have failed [2,[5][6][7][8][9][10][12][13][14]17,[17][18][19][20]22,23,26,27,29,[32][33][34][35][36].…”