MPI Correctness Checking with Marmot

Krammer, Bettina; Hilbrich, Tobias; Himmler, Valentin; Czink, Blasius; Dichev, Kiril; Müller, Matthias S.

doi:10.1007/978-3-540-68564-7_5

Cited by 6 publications

(6 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Related research has also tried to improve the efficiency of error diagnosis [26]- [28], such as by using a daemon process [25]. Many of these methods rely on the MPI environment [32]- [34]. Concerning the algorithm itself, researchers have proposed the hardware faulttolerant algorithm [29].…”

Section: Related Workmentioning

confidence: 99%

FT-PBLAS: PBLAS-Based Fault-Tolerant Linear Algebra Computation on High-performance Computing Systems

2020

View full text Add to dashboard Cite

As high-performance computing (HPC) systems have scaled up, resilience has become a great challenge. To guarantee resilience, various kinds of hardware and software techniques have been proposed. However, among popular software fault-tolerant techniques, both the checkpoint-restart approach and the replication technique face challenges of scalability in the era of peta-and exa-scale systems due to their numerous processes. In this situation, algorithm-based approaches, or algorithm-based fault tolerance (ABFT) mechanisms, have become attractive because they are efficient and lightweight. Although the ABFT technique is algorithm-dependent, it is possible to implement it at a low level (e.g., in libraries for basic numerical algorithms) and make it application-independent. However, previous ABFT approaches have mainly aimed at achieving fault tolerance in integrated circuits (ICs) or at the architecture level and are therefore not suitable for HPC systems; e.g., they use checksums of rows and columns of matrices rather than checksums of blocks to detect errors. Furthermore, they cannot deal with errors caused by node failure, which are common in current HPC systems. To solve these problems, this paper proposes FT-PBLAS, a PBLAS-based library for fault-tolerant parallel linear algebra computations that can be regarded as a fault-tolerant version of the parallel basic linear algebra subprograms (PBLAS), because it provides a series of fault-tolerant versions of interfaces in PBLAS. To support the underlying error detection and recovery mechanisms in the library, we propose a block-checksum approach for non-fatal errors and a scheme for addressing node failure, respectively. We evaluate two fault-tolerant mechanisms and FT-PBLAS on HPC systems, and the experimental results demonstrate the performance of our library. INDEX TERMS Algorithm-based fault tolerance, HPC systems, node failure, matrix multiplication, linear algebra computations.

show abstract

Section: Related Workmentioning

confidence: 99%

FT-PBLAS: PBLAS-Based Fault-Tolerant Linear Algebra Computation on High-performance Computing Systems

2020

View full text Add to dashboard Cite

show abstract

“…In its initial form, the basic output of MUST is an HTML table that follows the format of Marmot [7]. In Marmot checks had to be implemented for each MPI call, even for the same error conditions, leading to significant code duplication of any error reporting.…”

Section: Mustmentioning

confidence: 99%

MPI Runtime Error Detection with MUST: Advanced Error Reports

Protze

Hilbrich

Supinski

et al. 2013

Tools for High Performance Computing 2012

Self Cite

View full text Add to dashboard Cite

“…Approaches for runtime error detection include ISP [3], Marmot [4], and Umpire [5]. ISP offers type matching for point-to-point calls while no publication details the algorithm in use.…”

Section: Related Workmentioning

confidence: 99%

“…Marmot [4] provides various checks for argument restrictions and datatype commit state, which we adopt in MUST. These checksfall into the following classes:…”

Section: A Argument Restrictionsmentioning

confidence: 99%

Holistic Debugging of MPI Derived Datatypes

Protze

Hilbrich

Knüpfer

et al. 2012

2012 IEEE 26th International Parallel and Distributed Processing Symposium

Self Cite

View full text Add to dashboard Cite

Abstract-The Message Passing Interface (MPI) specifies an API that allows programmers to create efficient and scalable parallel applications. The standard defines multiple constraints for each function parameter. For performance reasons, no MPI implementation checks all of these constraints at runtime. Derived datatypes are an important concept of MPI and allow users to describe an application's data structures for efficient and convenient communication. Using existing infrastructure we present scalable algorithms to detect usage errors of basic and derived MPI datatypes. We detect errors that include constraints for construction and usage of derived datatypes, matching their type signatures in communication, and detecting erroneous overlaps of communication buffers.We implement these checks in the MUST runtime error detection framework. We provide a novel representation of error locations to highlight usage errors. Further, approaches to buffer overlap checking can cause unacceptable overheads for non-contiguous datatypes. We present an algorithm that uses patterns in derived MPI datatypes to avoid these overheads without losing precision. Application results for the benchmark suites SPEC MPI2007 and NAS Parallel Benchmarks for up to 2048 cores show that our approach applies to a broad range of applications and that our extended overlap check improves performance by two orders of magnitude. Finally, we augment our runtime error detection component with a debugger extension to support in-depth analysis of the errors that we find as well as semantic errors. This extension to gdb provides information about MPI datatype handles and enables gdb -and other debuggers based on gdb -to display the content of a buffer as used in MPI communications.

show abstract

MPI Correctness Checking with Marmot

Cited by 6 publications

References 14 publications

FT-PBLAS: PBLAS-Based Fault-Tolerant Linear Algebra Computation on High-performance Computing Systems

FT-PBLAS: PBLAS-Based Fault-Tolerant Linear Algebra Computation on High-performance Computing Systems

MPI Runtime Error Detection with MUST: Advanced Error Reports

Holistic Debugging of MPI Derived Datatypes

Contact Info

Product

Resources

About