Clock delta compression for scalable order-replay of non-deterministic parallel applications

Sato, Kento; Ahn, Dong H.; Laguna, Ignacio; Lee, Gregory L.; Schulz, Martin

doi:10.1145/2807591.2807642

Cited by 19 publications

(20 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As depicted in Figure 1, if an application is not deterministic, then external methods can be used to make it deterministic. For example, one can identify and fix races with a race detector such as Archer [4], or directly determinize an execution using a capture-playback framework such as ReMPI [33].…”

Section: Workflow For Multi-level Analysismentioning

confidence: 99%

Multi-Level Analysis of Compiler-Induced Variability and Performance Tradeoffs

Bentley

Briggs

Gopalakrishnan

et al. 2019

Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing

Self Cite

View full text Add to dashboard Cite

No Yes No Yes Create FLiT tests User Code Deterministic? Reproducibility and Performance Is the fastest repro sufficient? Done Library, Source, and Function Blame Debug Issue using standard tools Determinize Run FLiT Tests FLiT Bisect Done Figure 1: Multi-level workflow. Levels are (1) determine variability-inducing compilations, (2) analyze the space of reproducibility and performance, and (3) debug variability by identifying files and functions causing variability. ABSTRACTSuccessful HPC software applications are long-lived. When ported across machines and their compilers, these applications often produce different numerical results, many of which are unacceptable. Such variability is also a concern while optimizing the code more aggressively to gain performance. Efficient tools that help locate the program units (files and functions) within which most of the variability occurs are badly needed, both to plan for code ports and to root-cause errors due to variability when they happen in the field. In this work, we offer an enhanced version of the open-source testing framework FLiT to serve these roles. Key new features of FLiT include a suite of bisection algorithms that help locate the root causes of variability. Another added feature allows an analysis of the tradeoffs between performance and the degree of variability. Our new contributions also include a collection of case studies. Results on the MFEM finite-element library include variability/performance tradeoffs, and the identification of a (hitherto unknown) abnormal level of result-variability even under mild compiler optimizations. Results from studying the Laghos proxy application include identifying a significantly divergent floating-point result-variability and successful root-causing down to the problematic function over as little as 14 program executions. Finally, in an evaluation of 4,376 controlled injections of floating-point perturbations on the LULESH proxy application, we showed that the FLiT framework has 100% precision and recall in discovering the file and function locations of the injections all within an average of only 15 program executions.

show abstract

Section: Workflow For Multi-level Analysismentioning

confidence: 99%

Multi-Level Analysis of Compiler-Induced Variability and Performance Tradeoffs

Bentley

Briggs

Gopalakrishnan

et al. 2019

Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…MCB is implemented primarily using MPI's non-blocking point-to-point communication primitives, specifically making heavy use of non-blocking matching functions such as MPI_Testsome during particle exchange. This leads to significant nondeterminism which can be directly observed in terms of differing numerical outputs from run to run (Cleveland et al, 2013;Sato et al, 2015). MCB's executions can be decomposed into three distinct communication patterns (i.e.…”

Section: Monte Carlo Benchmarkmentioning

confidence: 99%

“…Tools for addressing some aspects of the nondeterministic problem have emerged, but they do not provide methods for systematically cataloging the nondeterminism in a given application. For example, record-and-replay (R&R) techniques (Chapp et al, 2018;Taufer et al, 2005) present an attractive method for mitigating the harmful aspects of nondeterminism in HPC applications-such as numerical irreproducibility (Chapp et al, 2015) and hampered debugging (Sato et al, 2015(Sato et al, , 2017)-but are restricted by their limited scalability (i.e. they cannot effectively record executions of HPC applications and the peta-and exascale).…”

Section: Introductionmentioning

confidence: 99%

A three-phase workflow for general and expressive representations of nondeterminism in HPC applications

Chapp

Rorabaugh

Sato

et al. 2019

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

Nondeterminism is an increasingly entrenched property of high-performance computing (HPC) applications and has recently been shown to seriously hamper debugging and reproducibility efforts. Tools for addressing the nondeterministic debugging problem have emerged, but they do not provide methods for systematically cataloging the nondeterminism in a given application. We propose a three-phase workflow for representing executions of nondeterministic message passing interface programs as event graphs, quantifying their structural similarity with graph kernels, and applying machine learning techniques to investigate shared properties across applications. We present an empirical study comparing two graph kernels’ suitability for this task and propose future uses of the methodology.

show abstract

“…While in many MPI programs, nondeterminism can be controlled by eliminating application sources of nondeterminism, such as calls to rand() and/or time(), in other programs this is difficult because of nondeterminism introduced by MPI point-to-point communication patterns. To address these applications, we rely on record-and-replay tools [39], [40], on which a fault-free run is recorded and it is then replayed in all subsequent faulty executions.…”

Section: B Parallel Tracing Overheadmentioning

confidence: 99%

FlipTracker: Understanding Natural Error Resilience in HPC Applications

Guo

Liu

Laguna

et al. 2018

SC18: International Conference for High Performance Computing, Networking, Storage and Analysis

Self Cite

View full text Add to dashboard Cite

As high-performance computing systems scale in size and computational power, the danger of silent errors, i.e., errors that can bypass hardware detection mechanisms and impact application state, grows dramatically. Consequently, applications running on HPC systems need to exhibit resilience to such errors. Previous work has found that, for certain codes, this resilience can come for free, i.e., some applications are naturally resilient, but few studies have shown the code patterns-combinations or sequences of computations-that make an application naturally resilient. In this paper, we present FlipTracker, a framework designed to extract these patterns using fine-grained tracking of error propagation and resilience properties, and we use it to present a set of computation patterns that are responsible for making representative HPC applications naturally resilient to errors. This not only enables a deeper understanding of resilience properties of these codes, but also can guide future application designs towards patterns with natural resilience.

show abstract

Clock delta compression for scalable order-replay of non-deterministic parallel applications

Cited by 19 publications

References 26 publications

Multi-Level Analysis of Compiler-Induced Variability and Performance Tradeoffs

Multi-Level Analysis of Compiler-Induced Variability and Performance Tradeoffs

A three-phase workflow for general and expressive representations of nondeterminism in HPC applications

FlipTracker: Understanding Natural Error Resilience in HPC Applications

Contact Info

Product

Resources

About