Formal verification of practical MPI programs

Abstract

This paper considers the problem of formal verification of MPI programs operating under a fixed test harness for safety properties without building verification models. In our approach, we directly model-check the MPI/C source code, executing its interleavings with the help of a verification scheduler. Unfortunately, the total feasible number of interleavings is exponential, and impractical to examine even for our modest goals. Our earlier publications formalized and implemented a partial order reduction approach that avoided exploring equivalent interleavings, and presented a verification tool called ISP. This paper presents algorithmic and engineering innovations to ISP, including the use of OpenMP parallelization, that now enables it to handle practical MPI programs, including: (i) ParMETIS -a widely used hypergraph partitioner, and (ii) MADRE -a Memory Aware Data Re-distribution Engine, both developed outside our group. Over these benchmarks, ISP has automatically verified up to 14K lines of MPI/C code, producing error traces of deadlocks and assertion violations within seconds.

Calculation coverage testing in scientific applications

“…In fact, several model-checking approaches have been applied in HPC, although their main focus is to verify the correctness of the MPI-based parallel/distributed program design [31,25,33,32,34,30,40,41]. Previous studies using the SPIN model checker [17] found it difficulty to create an accurate model of the program being verified [25,33,32,34].…”

A Robust and Efficient Message Passing Library for Volunteer Computing Environments

“…However, some features allow nondeterministic behavior [48]. Most notable, the ability to specify wildcards on the receiver side of a point-to-point operation (MPI_ANY_SOURCE), or functions allowing to complete a subset of pending communication operations such as MPI_Waitany or MPI_Waitsome.…”

The pilot approach to cluster programming in C

“…The data type is MPI_BYTE (Pilot %b). The number of iterations for each data point starts at 1000, and begins to decrease (by halves) as the message size passes 2 15 . With Pilot's level 0 of checking, running on an HP Opteron (2.2 GHz) cluster with Myrinet-2G (GM) interconnect, using HP Linux XC 3.1 and HP-MPI, it is clear that the time for any Pilot round is only slightly more than for pure MPI, with more noticeable overhead for smaller message sizes, as would be expected.…”

“…Since Pilot is based on CSP, it should be possible to carry out formal verification on a Pilot program, and uncover potential deadlock scenarios without actually running it. Such work has been carried out for MPI programs-for example, Vo et al [15]-so the Pilot case ought to be simpler.…”