Identifying Degree and Sources of Non-Determinism in MPI Applications Via Graph Kernels

Abstract

As the scientific community prepares to deploy an increasingly complex and diverse set of applications on exascale platforms, the need to assess reproducibility of simulations and identify the root causes of reproducibility failures increases correspondingly. One of the greatest challenges facing reproducibility issues at exascale is the inherent non-determinism at the level of inter-process communication. The use of non-deterministic communication constructs are necessary to boost performance, but communication non-determinism can also hamper software correctness and result reproducibility.To address this challenge, we propose a software framework for identifying the percentage and sources of communication nondeterminism. We model parallel executions as directed graphs and leverage graph kernels to characterize run-to-run variations in interprocess communication. We demonstrate the effectiveness of graph kernel similarity as a proxy for non-determinism, by showing that these kernels can quantify the type and degree of non-determinism present in communication patterns. To demonstrate our framework's ability to link and quantify runtime non-determinism to root sources, we demonstrate results for an adaptive mesh refinement application, where our framework automatically quantifies the impact of function calls on nondeterminism, and a Monte Carlo application, where our framework automatically quantifies the impact of parameter configurations on nondeterminism.

A Survey of Graph Comparison Methods with Applications to Nondeterminism in High-Performance Computing

“…In ANACIN-X (Chapp et al, 2021), we demonstrate that quantitative comparison using graph kernels can indeed be an appropriate proxy for nondeterminism and can further be used to identify the functions of the root cause of nondeterminism. ANACIN-X takes as input execution traces for a nondeterministic synthetic application using the DUMPI tracing library Wilke and transforms them into event graphs.…”

“…A rare case of graph kernel application to nondeterminism ANACIN-X (Chapp et al, 2021) is a rare case of application of graph kernel for identifying the percentage and sources of communication nondeterminism. The software framework models parallel executions as directed graphs and leverage graph kernels to characterize run-to-run variations in interprocess communication.…”

“…In recent years, increasing awareness of the need to manage nondeterminism in parallel applications (Gopalakrishnan et al, 2017) has given rise to a few software solutions. Three important solutions most closely address the problem: PopMine (Seo et al, 2011), SABALAN (Alimadadi et al, 2018), and ANACIN-X (Chapp et al, 2021). PopMine Seo et al (2011) is a tool that analyzes traces of message orders in MPI applications, also represented by an event graph-like structure to determine a minimal DAG that triggers a given bug.…”

“…The event graphs are subsequently compared using the Weisfeiler-Lehman graph kernel. Figure 10 shows this comparison for the miniAMR applications presented in Chapp et al (2021).…”

“…The convergence of extreme hardware concurrency and the effective overlap of computation and communication in asynchronous executions are resulting in growing nondeterminism in High-Performance Computing (HPC) applications, as illustrated in Figure 1 and presented in Ahn et al (2013); Gopalakrishnan et al (2017); Sato et al (2017); Chapp et al (2015Chapp et al ( , 2018Chapp et al ( , 2021. Nondeterminism can manifest at multiple levels in the software stack: it can manifest in lowlevel communication primitives (e.g., the inherent nondeterminism of nonblocking matching functions in MPI); it can manifest in libraries (e.g., dynamic load-balancing libraries), as presented in Lusk et al (2015); or it can display at the application level (e.g., Monte-Carlo simulations).…”