Resilience for Stencil Computations with Latent Errors

Abstract

Projections and measurements of error rates in near-exascale and exascale systems suggest a dramatic growth, due to extreme scale (10 9 cores), concurrency, software complexity, and deep submicron transistor scaling. Such a growth makes resilience a critical concern, and may increase the incidence of errors that "escape", silently corrupting application state. Such errors can often be revealed by application software tests but with long latencies, and thus are known as latent errors. We explore how to efficiently recover from latent errors, with an approach called application-based focused recovery (ABFR). Specifically we present a case study of stencil computations, a widely useful computational structure, showing how ABFR focuses recovery effort where needed, using intelligent testing and pruning to reduce recovery effort, and enables recovery effort to be overlapped with application computation. We analyze and characterize the ABFR approach on stencils, creating a performance model parameterized by error rate and detection interval (latency). We compare projections from the model to experimental results with the Chombo stencil application, validating the model and showing that ABFR on stencil can achieve a significant reductions in error recovery cost (up to 400x) and recovery latency (up to 4x). Such reductions enable efficient execution at scale with high latent error rates.

Algorithm-Based Fault Tolerance for Parallel Stencil Computations

“…This will allow to recover from the state of the domain at that point in case of another error. Recall that an error correction mechanism specifically dedicated to stencil computations has already been presented [18], wherein authors exploit stencil locality to reduce the re-execution cost after an error has been detected. Such a method could be used to further lower the cost of re-computation in case of errors.…”

Resilient N-Body Tree Computations with Algorithm-Based Focused Recovery: Model and Performance Analysis

“…Recently, we have successfully applied ABFR to stencil computations [4], which are perfectly suited to ABFR due to their regular and neighborbased communication pattern. The tree-based propagation pattern of N-Body computations is much more challenging for ABFR.…”

“…We propose to use the Algorithm-Based Focused Recovery (ABFR) approach [4] for N-body computations. ABFR exploits application semantics and versioned states to bound error impact and further localize recovery.…”

“…In previous work [4], we proposed a new approach, Application-Based Focused Recovery (ABFR), that exploits application data flow and intermediate states to focus recovery on an accurate estimate of potentially corrupted data. Our study on stencil computations demonstrated ABFR reduces recovery cost by up to 400x.…”

Application health monitoring for extreme‐scale resiliency using cooperative fault management

“…Techniques for detecting (and in some cases correcting) silent errors have been studied for a variety of iterative solvers, [31][32][33][34][35] adaptive numerical integrators, [36][37][38] and other widely used computational approaches. 5,39,40 Clearly, not all application outputs would be useful as health indicators, but our experience, informed by discussions with many colleagues, is that most computational scientists develop techniques to check their simulations to determine whether the results are believable. Many such checks could be automated and applied throughout a simulation using our approach.…”