Using Migratable Objects to Enhance Fault Tolerance Schemes in Supercomputers

Abstract

Abstract-Supercomputers have seen an exponential increase in their size in the last two decades. Such a high growth rate is expected to take us to exascale in the timeframe 2018-2022. But, to bring a productive exascale environment about, it is necessary to focus on several key challenges. One of those challenges is fault tolerance. Machines at extreme scale will experience frequent failures and will require the system to avoid or overcome those failures. Various techniques have recently been developed to tolerate failures. The impact of these techniques and their scalability can be substantially enhanced by a parallel programming model called migratable objects. In this paper, we demonstrate how the migratable-objects model facilitates and improves several fault tolerance approaches. Our experimental results on thousands of cores suggest fault tolerance schemes based on migratable objects have low performance overhead and high scalability. Additionally, we present a performance model that predicts a significant benefit of using migratable objects to provide fault tolerance at extreme scale.

A Fault-Tolerance Protocol for Parallel Applications with Communication Imbalance

“…The checkpoint buddy of the failed node will provide a checkpoint to the replacement node. This approach has been shown to be scalable and it can recover from most failures in real HPC systems [10]. Other variants of checkpoint/restart create a multilevel framework where checkpoints can go to local or shared storage [7].…”

“…An enhancement to checkpoint/restart is message logging [15], a technique that stores checkpoints and, in principle, stores all the messages in an execution. There are multiple implementations of message logging for HPC systems [10], [16], [17]. The benefit of storing communication is that a failure only requires the failed node to rollback, hence only local rollback is needed.…”

“…The performance overhead of those strategies can be kept low [11], [12], they feature a very efficient energy profile [8], [9], and they make possible to parallelize recovery [10]. To leverage all those features, it is imperative to address the major drawback of message logging, namely its increase in memory footprint.…”

“…A failure in the system will only roll back the failed nodes, and have the rest of the system replay the stored messages to recover the restarted nodes. Message logging has been demonstrated to save energy [8], [9], reduce recovery time [10], and have a small overhead [11], [12]. The main drawback of message logging is its potentially large memory footprint due to the message log.…”

A Minimally Intrusive Low-Memory Approach to Resilience for Existing Transient Solvers

“…In contrast, the Cray XK6/XK7 (Titan) at Oak Ridge National Laboratory (10-20/27 petaflops) achieves a MTBI of 132/173 h [2]. The anticipated failure rate of an exascale machine is likely to be higher than present systems [8,9,23,28] and therefore application resilience is critical in maintaining the usefulness of any future exascale system.…”