A Scalable and Distributed Dynamic Formal Verifier for MPI Programs

Abstract

Abstract-Standard testing methods of MPI programs do not guarantee coverage of all non-deterministic interactions (e.g., wildcard-receives). Programs tested by these methods can have untested paths (bugs) that may become manifest unexpectedly. Previous formal dynamic verifiers cover the space of non-determinism but do not scale, even for small applications. We present DAMPI, the first dynamic analyzer for MPI programs that guarantees scalable coverage of the space of non-determinism through a decentralized algorithm based on Lamport-clocks. DAMPI computes alternative non-deterministic matches and enforces them in subsequent program replays. To avoid interleaving explosion, DAMPI employs heuristics to focus coverage to regions of interest. We show that DAMPI can detect deadlocks and resource-leaks in real applications. Our results on a wide range of applications using over a thousand processes, which is an order of magnitude larger than any previously reported results for MPI dynamic verification tools, demonstrate that DAMPI provides scalable, user-configurable testing coverage.

A Sound Reduction of Persistent-Sets for Deadlock Detection in MPI Applications

“…MSPOE will, on the other hand, examine the initial trace, and perform macroscopic commutation aware persistent set reductions. This is a search bounding method substantially different from other obvious reduction approaches (e.g., depth-bounding or bounded mixing [15]), and further this bounding heuristic is tuned toward detecting orphaning deadlocks. Further studies are underway to further characterize MSPOE.…”

“…a safety property). Formal dynamic verifiers such as ISP [14,17] and DAMPI [15,16] take an approach that integrates the best features of testing tools (ability to run on user applications) and model checking (message match non-determinism coverage guarantees). They run the MPI program under the control of verification-oriented scheduling mechanisms (a central scheduler for ISP and logical clocks for DAMPI).…”

Combining static and dynamic validation of MPI collective communications

“…Dealing with dynamic tools able to check collective operations, we can mention DAMPI [13], Marmot [6,8], Umpire [12,8], MPI-CHECK [7,8], Intel Message Checker (IMC) [2,8] and MUST [5,4]. Umpire, Marmot and MUST rely on a dynamic analysis of MPI calls instrumented through the MPI profiling interface (PMPI).…”

Multiparty Session C: Safe Parallel Programming with Message Optimisation

“…In order to reduce the state space of possible thread interleavings of an execution, the tool exploits an independence between thread actions. Later in [35], they improved its scheduling policy to gain efficiency of the verification. TASS [32] is another model checking-based tool for a deadlock analysis in MPI.…”