Dynamic Verification of MPI Programs with Reductions in Presence of Split Operations and Relaxed Orderings

Abstract

Abstract. Dynamic verification methods are the natural choice for debugging real world programs when model extraction and maintenance are expensive. Message passing programs written using the MPI library fall under this category. Partial order reduction can be very effective for MPI programs because for each process, all its local computational steps, as well as many of its MPI calls, commute with the corresponding steps of all other processes. However, when dependencies arise among MPI calls, they are often a function of the runtime state. While this suggests the use of dynamic partial order reduction (DPOR), three aspects of MPI make previous DPOR algorithms inapplicable: (i) many MPI calls are allowed to complete out of program order; (ii) MPI has global synchronization operations (e.g., barrier) that have a special weak semantics; and (iii) the runtime of MPI cannot, without intrusive modifications, be forced to pursue a specific interleaving because of MPI's liberal message matching rules, especially pertaining to 'wildcard receives'. We describe our new dynamic verification algorithm 'POE' that exploits the out of order completion semantics of MPI by delaying the issuance of MPI calls, issuing them only according to the formation of match-sets, which are ample 'big-step' moves. POE guarantees to manifest any feasible interleaving by dynamically rewriting wildcard receives by specific-source receives. This is the first dynamic model-checking algorithm with reductions for (a large subset of) MPI that guarantees to catch all deadlocks and local assertion violations, and is found to work well in practice.

TransDPOR: A Novel Dynamic Partial-Order Reduction Technique for Testing Actor Programs

“…Specifically, MPI processes assume FIFO channels and usually have matched sends and receives. Vakkalanka et al [26] proposed a stateless DPOR technique for MPI programs, called POE, which exploits these constraints. POE can produce only one interleaving for large MPI programs that do not have an MPI wild-card receive.…”

“…Partial-order reduction (POR) techniques [5,6,8,9,11,20,22,[24][25][26][27]29] can be applied to help mitigate the resulting state-space explosion by exploring a representative subset of all possible interleavings. POR techniques have been widely used for testing and verification of concurrent protocols and software, including in tools such as SPIN [13], VeriSoft [8], and Java PathFinder [28].…”

A Sound Reduction of Persistent-Sets for Deadlock Detection in MPI Applications

“…Readers are encouraged to refer [14] for more details. We have presented only the required rules and details in order to make the paper self-contained.…”

“…a safety property). Formal dynamic verifiers such as ISP [14,17] and DAMPI [15,16] take an approach that integrates the best features of testing tools (ability to run on user applications) and model checking (message match non-determinism coverage guarantees). They run the MPI program under the control of verification-oriented scheduling mechanisms (a central scheduler for ISP and logical clocks for DAMPI).…”

“…We define a notion of orphaning deadlocks in which an MPI receive is left without a matching send in some MPI program execution state. We modify ISP's dynamic partial order reduction algorithm called POE (standing for Partial Order avoiding Elusive interleavings, [14]) to result in a new algorithm called MSPOE (MacroScopic POE). MSPOE applies to MPI programs that "do not decode data," i.e., do not employ data dependent control flows, and do not alter their control flows based on which sends a non-deterministic receive matches with.…”