SReplay

Abstract

Replay of parallel execution is required by HPC debuggers and resilience mechanisms. Up-to-date, there is no existing deterministic replay solution for one-sided communication. The essential problem is that the readers of updated data do not have any information on which remote threads produced the updates, the conventional happens-before based ordering tracking techniques are challenging to work at scale. This paper presents SReplay, the first software tool for sub-group deterministic record and replay for one-sided communication. SReplay allows the user to specify and record the execution of a set of threads of interest (sub-group), and then deterministically replays the execution of the sub-group on a local machine without starting the remaining threads. SReplay ensures sub-group determinism using a hybrid data-and order-replay technique. SReplay maintains scalability by a combination of local logging and approximative event order tracking within sub-group. Our evaluation on deterministic and nondeterministic UPC programs shows that SReplay introduces an overhead ranging from 1.3× to 29×, when running on 1,024 cores and tracking up to 16 threads.Figure 1: Two-sided and One-sided Communication.The essence of one-sided communication is its implicit nature. In MPI, a typical communication involves MPI_Send/MPI_Recv pairs, which carries both data transfer and synchronization semantic and the initiating task can be determined in the receive operation (shown in Figure 1 (a)). Also, the memory location involved in the communication is visible only to one rank. These features made MPI communication easy to intercept at runtime. In one-sided communication (shown in Figure 1 (b)), a thread could write to any remote memory location in shared address space by a Put without notifying others. Later, a thread could read the new value produced by that earlier writer using a local access (since the thread is affiliated with the shared memory module), the reader is not aware of the thread which previously produced the value. Such implicit one-sided communication removes the implicit synchronization between sender and receiver in message passing and potentially offers better performance. However, this advantage comes at the price of nondeterminism and complex debugging because previous techniques based on dependence tracking could not apply. Deterministic Record and ReplayDeterministic Record and Replay (R&R) consists of monitoring the execution of a multithreaded application on a parallel machine, and exactly reproducing this execution later. R&R requires recording in a log all nondeterministic events that occurred during the initial execution. They include the inputs to the execution (e.g., return values from system calls) and the order of the inter-thread communications (e.g., the interleaving of the inter-thread data dependences). During the replay phase, the logged inputs are fed into to the execution at the correct times, and the memory accesses are forced to interleave according to the log.

A Survey of Graph Comparison Methods with Applications to Nondeterminism in High-Performance Computing

“…State-of-the-art record-and-replay tools such as ReMPI (Sato et al, 2015) target production-scale runs and prioritize scalability in terms of runtime and record size. Other record-and-replay tools target hybrid MPI + OpenMP executions (Budanur et al, 2012), MPI applications using one-sided communication (Qian et al, 2016b,a), replay of isolated subgroups of processes (Xue et al, 2009), and probabilistic replay (Park et al, 2009). In addition, tools such as NINJA (Sato et al, 2017) are used in conjunction with record-and-replay tools to improve the chances of capturing nondeterministic bugs.…”

Record-and-Replay Techniques for HPC Systems: A Survey

“…MPI's onesided communication routines in particular pose unique challenges to R&R. Quian et al proposed two techniques for addressing this challenge-OPR [42] and its successor SReplay [43]. SReplay proposes a hybrid-replay scheme which permits replay of subgroups of processes.…”