NR-MPI: A Non-stop and Fault Resilient MPI

Suo, Guang; Lu, Yutong; Liao, Xiangke; Xie, Min; Cao, Hongjia

doi:10.1109/icpads.2013.37

Cited by 13 publications

(9 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Despite representing by itself a key contribution, the FT-MPI standard was deprecated. Other works, including [10,22,23] followed the same approach defining MPI primitives that allow the application to survive faults.…”

Section: Related Workmentioning

confidence: 99%

“…Another strategy based on FT-MPI has been proposed: Non-Stop and Fault-Resilient MPI (NR-MPI) [23]. NR-MPI implements the semantics of FT-MPI on MPICH, one of the most widely used MPI implementations [24].…”

Section: Related Workmentioning

confidence: 99%

“…Then, each process finds a partner within the cluster using an exclusive or operation on its rank (line 15). The lowest rank process sends the list with numbers greater than the pivot to the partner with a higher rank and receives the list of numbers less than the pivot from the partner (16)(17)(18)(19)(20)(21)(22)(23). After this exchange of lists, each process merges the received list with its list number that was not exchanged (line 19 and 23).…”

mentioning

confidence: 99%

See 2 more Smart Citations

Running resilient MPI applications on a Dynamic Group of Recommended Processes

Camargo

Duarte

2018

J Braz Comput Soc

View full text Add to dashboard Cite

High-performance computing systems run applications that can take several hours to execute and have to deal with the occurrence of a potentially large number of faults. Most of the existing fault-tolerant strategies for these systems assume crash faults that are permanent events are easily detected. This is not the case in several real systems, in particular in shared clusters, in which even the load variation may cause performance problems that are virtually equivalent to faults. In this work, we present a new model to deal with this problem in which processes execute tests among themselves in order to determine whether the processors (or cores) on which they are running are recommended or non-recommended. Processes classified as recommended form a Dynamic Group of Recommended Processes (DGRP) that runs the application. The DGRP is formed only by processes that have not been tested as non-recommended by all DGRP processes. A process not in the DGRP that is continuously tested as recommended can rejoin the DGRP after a round of consensus executed by DGRP processes. Experimental results are presented obtained from a MPI-based implementation in which the HyperQuickSort parallel sorting algorithm reconfigures itself at runtime to tolerate up to N − 1 faults (in a system with N processes) while sorting up to 1 billion integers.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Running resilient MPI applications on a Dynamic Group of Recommended Processes

Camargo

Duarte

2018

J Braz Comput Soc

View full text Add to dashboard Cite

show abstract

“…In our previous work [11], we all also present NR-MPI, which is implemented based on MPICH. The failure recovery of MPI library is based on our own fault tolerant MPI constructs.…”

Section: Fault Tolerant Techniquesmentioning

confidence: 99%

NR-MPI: A N on-stop and Fault Resilient MPI SupportingProgrammer Defined Data Backup and Restore for E-scaleSuper Computing Systems

Suo

Liao

et al. 2016

JSFI

View full text Add to dashboard Cite

Fault resilience has became a major issue for HPC systems, particularly, in the perspective of future E-scale systems, which will consist of millions of CPU cores and other components. MPI-level fault tolerant constructs, such as ULFM, are being proposed to support software level fault tolerance. However, there are few systematic evaluations by application programmers using benchmarks or pseudo applications. This paper proposes NR-MPI, a N on-stop and Fault Resilient MPI, supporting programmer defined data backup and restore. To help programmers write fault tolerant programs, NR-MPI provides a set of friendly programming interfaces and a state transition diagram for data backup and restore. This paper focuses on design, implementation and evaluation of NR-MPI. Specifically,this paper puts emphases on failure detection in MPI library, friendly programming interface extending for NR-MPI and examples of fault tolerant programs based NR-MPI. Furthermore, to support failure recovery of applications, NR-MPI implements data backup interfaces based on double in-memory checkpoint/restart. We conduct experiments with both NPB benchmarks and Sweep3D on TH supercomputer in NSCC-TJ. Experimental results show that NR-MPI based fault tolerant programs can recover from failures online without restarting, and the overhead is small even for applications with tens of thousands of cores.

show abstract

“…NR-MPI (Non-stop Fault Resilient MPI) [12] implements the semantics of FT-MPI and supports online recovery of MPI_COMM_WORLD. NR-MPI exploits the RMS (Resource Management System, i.e.…”

Section: Related Workmentioning

confidence: 99%

Fault Tolerance Assistant (FTA): An Exception Handling Programming Model for MPI Applications

Fang

Laguna

Sato

et al. 2016

View full text Add to dashboard Cite

Future high-performance computing systems may face frequent failures with their rapid increase in scale and complexity. Resilience to faults has become a major challenge for large-scale applications running on supercomputers, which demands fault tolerance support for prevalent MPI applications. Among failure scenarios, process failures are one of the most severe issues as they usually lead to termination of applications. However, the widely used MPI implementations do not provide mechanisms for fault tolerance. We propose FTA-MPI (Fault Tolerance Assistant MPI), a programming model that provides support for failure detection, failure notification and recovery. Specifically, FTA-MPI exploits a try/catch model that enables failure localization and transparent recovery of process failures in MPI applications. We demonstrate FTA-MPI with synthetic applications and a molecular dynamics code CoMD, and show that FTA-MPI provides high programmability for users and enables convenient and flexible recovery of process failures.

show abstract

NR-MPI: A Non-stop and Fault Resilient MPI

Cited by 13 publications

References 20 publications

Running resilient MPI applications on a Dynamic Group of Recommended Processes

Running resilient MPI applications on a Dynamic Group of Recommended Processes

NR-MPI: A N on-stop and Fault Resilient MPI SupportingProgrammer Defined Data Backup and Restore for E-scaleSuper Computing Systems

Fault Tolerance Assistant (FTA): An Exception Handling Programming Model for MPI Applications

Contact Info

Product

Resources

About