Checkpointing strategies for parallel jobs

Abstract

This work provides an analysis of checkpointing strategies for minimizing expected job execution times in an environment that is subject to processor failures. In the case of both sequential and parallel jobs, we give the optimal solution for exponentially distributed failure inter-arrival times, which, to the best of our knowledge, is the first rigorous proof that periodic checkpointing is optimal. For non-exponentially distributed failures, we develop a dynamic programming algorithm to maximize the amount of work completed before the next failure, which provides a good heuristic for minimizing the expected execution time. Our work considers various models of job parallelism and of parallel checkpointing overhead. We first perform extensive simulation experiments assuming that failures follow Exponential or Weibull distributions, the latter being more representative of real-world systems. The obtained results not only corroborate our theoretical findings, but also show that our dynamic programming algorithm significantly outperforms previously proposed solutions in the case of Weibull failures. We then discuss results from simulation experiments that use failure logs from production clusters. These results confirm that our dynamic programming algorithm significantly outperforms existing solutions for real-world clusters.

On the Viability of Compression for Reducing the Overheads of Checkpoint/Restart-Based Fault Tolerance

“…Therefore, in the optimal case, the number of checkpoints equals the number of failures, which equals the number of recoveries. There are various works that define optimal checkpoint intervals [28], [29]. Finally, we assume that checkpoint commit is synchronous; that is, the primary application process is paused during the commit operation and is not resumed until checkpoint commit is complete.…”

Resilient co-scheduling of malleable applications

“…The first step is to generate a fault distribution: we use an existing fault simulator developed in [21,22]. In our case, we use this simulator with an exponential law of parameter λ.…”

Composing resilience techniques: ABFT, periodic and incremental checkpointing

“…Many models are available to understand the behavior of checkpoint/restart [19,20,21,22], and thereby to define an optimal checkpoint period. [23] proposes a scalability model to evaluate the impact of failures on application performance.…”

“…, t j , and to checkpoint after t j , without any intermediate checkpoint, and knowing that a checkpoint has been taken after task t i−1 . To the best of our knowledge, the expectation E(W, C) of the time needed to successfully compute during W seconds and then take a checkpoint of duration C is known only for Exponentially distributed failures; from [22], we know that:…”