Scalable Resilience Against Node Failures for Communication-Hiding Preconditioned Conjugate Gradient and Conjugate Residual Methods

Levonyak, Markus; Pacher, Christina; Gansterer, Wilfried N.

doi:10.1137/1.9781611976137.8

Cited by 5 publications

(2 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast to restarting, a number of algorithm-based recovery strategies have been proposed, including approximate or heuristic interpolation methods (Agullo et al, 2016a, 2016b). An approach of exactly recovering the state of the iterative solver before the node failure has been investigated for the Preconditioned Conjugate Gradient (PCG) and related methods (Levonyak et al, 2020; Pachajoa et al, 2018). This also includes studying scenarios with multiple simultaneous node failures (Pachajoa et al, 2019) and scenarios where no replacement nodes are available (Pachajoa et al, 2019).…”

Section: Numerical Algorithms For Resiliencementioning

confidence: 99%

Resiliency in numerical algorithm design for extreme scale simulations

Agullo

Altenbernd

Anzt

et al. 2021

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

This work is based on the seminar titled ‘Resiliency in Numerical Algorithm Design for Extreme Scale Simulations’ held March 1–6, 2020, at Schloss Dagstuhl, that was attended by all the authors. Advanced supercomputing is characterized by very high computation speeds at the cost of involving an enormous amount of resources and costs. A typical large-scale computation running for 48 h on a system consuming 20 MW, as predicted for exascale systems, would consume a million kWh, corresponding to about 100k Euro in energy cost for executing 1023 floating-point operations. It is clearly unacceptable to lose the whole computation if any of the several million parallel processes fails during the execution. Moreover, if a single operation suffers from a bit-flip error, should the whole computation be declared invalid? What about the notion of reproducibility itself: should this core paradigm of science be revised and refined for results that are obtained by large-scale simulation? Naive versions of conventional resilience techniques will not scale to the exascale regime: with a main memory footprint of tens of Petabytes, synchronously writing checkpoint data all the way to background storage at frequent intervals will create intolerable overheads in runtime and energy consumption. Forecasts show that the mean time between failures could be lower than the time to recover from such a checkpoint, so that large calculations at scale might not make any progress if robust alternatives are not investigated. More advanced resilience techniques must be devised. The key may lie in exploiting both advanced system features as well as specific application knowledge. Research will face two essential questions: (1) what are the reliability requirements for a particular computation and (2) how do we best design the algorithms and software to meet these requirements? While the analysis of use cases can help understand the particular reliability requirements, the construction of remedies is currently wide open. One avenue would be to refine and improve on system- or application-level checkpointing and rollback strategies in the case an error is detected. Developers might use fault notification interfaces and flexible runtime systems to respond to node failures in an application-dependent fashion. Novel numerical algorithms or more stochastic computational approaches may be required to meet accuracy requirements in the face of undetectable soft errors. These ideas constituted an essential topic of the seminar. The goal of this Dagstuhl Seminar was to bring together a diverse group of scientists with expertise in exascale computing to discuss novel ways to make applications resilient against detected and undetected faults. In particular, participants explored the role that algorithms and applications play in the holistic approach needed to tackle this challenge. This article gathers a broad range of perspectives on the role of algorithms, applications and systems in achieving resilience for extreme scale simulations. The ultimate goal is to spark novel ideas and encourage the development of concrete solutions for achieving such resilience holistically.

show abstract

Section: Numerical Algorithms For Resiliencementioning

confidence: 99%

Resiliency in numerical algorithm design for extreme scale simulations

Agullo

Altenbernd

Anzt

et al. 2021

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

show abstract

“…In [16], Levonyak et al extend the concept of ESR to the pipelined PCG algorithm, while maintaining its communication-hiding properties.…”

Section: Related Workmentioning

confidence: 99%

Algorithm-Based Checkpoint-Recovery for the Conjugate Gradient Method

Pachajoa

Pacher

Levonyak

et al. 2020

49th International Conference on Parallel Processing - ICPP

Self Cite

View full text Add to dashboard Cite

As computers reach exascale and beyond, the incidence of faults will increase. Solutions to this problem are an active research topic. We focus on strategies to make the preconditioned conjugate gradient (PCG) solver resilient against node failures, speci cally, the exact state reconstruction (ESR) method, which exploits redundancies in PCG. Reducing the frequency at which redundant information is stored lessens the runtime overhead. However, after the node failure, the solver must restart from the last iteration for which redundant information was stored, which increases recovery overhead. This formulation highlights the method's similarities to checkpoint-restart (CR). Thus, this method, which we call ESR with periodic storage (ESRP), can be considered a form of algorithm-based checkpointrestart. The state is stored implicitly, by exploiting redundancy inherent to the algorithm, rather than explicitly as in CR. We also minimize the amount of data to be stored and retrieved compared to CR, but additional computation is required to reconstruct the solver's state. In this paper, we describe the necessary modi cations to ESR to convert it into ESRP, and perform an experimental evaluation. We compare ESRP experimentally with previously-existing ESR and application-level in-memory CR. Our results con rm that the overhead for ESR is reduced signi cantly, both in the failure-free case, and if node failures are introduced. In the former case, the overhead of ESRP is usually lower than that of CR. However, CR is faster if node failures happen. We claim that these di erences can be alleviated by the implementation of more appropriate preconditioners.

show abstract