Exploring versioned distributed arrays for resilience in scientific applications

Abstract

Exascale studies project reliability challenges for future HPC systems. We present the Global View Resilience (GVR) system, a library for portable resilience. GVR begins with a subset of the Global Arrays interface, and adds new capabilities to create versions, name versions, and compute on version data. Applications can focus versioning where and when it is most productive, and customize for each application structure independently. This control is portable, and its embedding in application source makes it natural to express and easy to maintain. The ability to name multiple versions and ''partially materialize'' them efficiently makes ambitious forward-recovery based on ''data slices'' across versions or data structures both easy to express and efficient. Using several large applications (OpenMC, preconditioned conjugate gradient (PCG) solver, ddcMD, and Chombo), we evaluate the programming effort to add resilience. The required changes are small (\ 2% lines of code (LOC)), localized and machine-independent, and perhaps most important, require no software architecture changes. We also measure the overhead of adding GVR versioning and show that overheads \ 2% are generally achieved. This overhead suggests that GVR can be implemented in large-scale codes and support portable error recovery with modest investment and runtime impact. Our results are drawn from both IBM BG/Q and Cray XC30 experiments, demonstrating portability. We also present two case studies of flexible error recovery, illustrating how GVR can be used for multi-version rollback recovery, and several different forward-recovery schemes. GVR's multi-version enables applications to survive latent errors (silent data corruption) with significant detection latency, and forward recovery can make that recovery extremely efficient. Our results suggest that GVR is scalable, portable, and efficient. GVR interfaces are flexible, supporting a variety of recovery schemes, and altogether GVR embodies a gentleslope path to tolerate growing error rates in future extreme-scale systems.

Resilience Design Patterns: A Structured Approach to Resilience at Extreme Scale (V.2.0)

“…Complementary to approaches that focus on resiliency of computational blocks, the Global View Resilience (GVR) project [47] concentrates on application data and guarantees resilience through multiple snapshot versions of the data whose creation is controlled by the programmer through application annotations. Bridges et al [36] proposed a malloc_failable that uses a callback mechanism to handle memory failures on dynamically allocated memory, so that the application programmer can specify recovery actions.…”

“…System-level solutions, such asDistributed MultiThreaded CheckPointing (DMTCP) [18], support transparent state saving and restoration using OS support. GVR [47] is a runtime system that provides fault tolerance to applications by versioning distributed arrays for checkpoint recovery, while the checkpoint-on-failure protocol [18] for MPI applications leverages the features of a high-quality fault-tolerant MPI implementation. In either case, algorithm-specific knowledge is needed to perform checkpoint recovery, Some ABFT solutions [122] can utilize the original or previously saved data as a replacement for lost or erroneous data and recover their state to the point at which the error/failure event occurred.…”

“…• GVR [47] is a runtime system that provides fault tolerance to applications by versioning distributed arrays. It supports checkpoint recovery based on application-specified mechanisms.…”

Resiliency in numerical algorithm design for extreme scale simulations

“…Global View Resilience (GVR) (Chien et al, 2017) accommodates APIs to enable multiple versioning of global arrays for the single program, multiple data programming model. The core idea is the fact that naive data redundancy approaches potentially store wrong applications states due to the large latency associated with error detection and notification.…”

Resilience Design Patterns: A Structured Approach to Resilience at Extreme Scale

“…In the context of HPC systems, software solutions typically implement roll-forward recovery using algorithm-specific knowledge. For example, Global View of Resilience (GVR) [13] uses versioning of distributed arrays supports, in which roll-forward recovery is based on application-specified mechanisms for each array structure.…”