A network-failure-tolerant message-passing system for terascale clusters

Graham, Richard L.; Choi, Sung‐Eun; Daniel, David; Desai, Nehal; Minnich, Ronald G.; Rasmussen, C. E.; Risinger, L. Dean; Sukalski, Mitchel W.

doi:10.1145/514191.514205

Cited by 31 publications

(9 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Graham et al provide a detailed description of the problem and have introduced LA-MPI in order to take advantage of network-based fault-tolerance [18]. Network fault-tolerance solutions, such as LA-MPI, complement our work.…”

Section: Related Workmentioning

confidence: 57%

A fault-tolerant strategy for virtualized HPC clusters

Walters

Chaudhary

2008

J Supercomput

View full text Add to dashboard Cite

Virtualization is a common strategy for improving the utilization of existing computing resources, particularly within data centers. However, its use for high performance computing (HPC) applications is currently limited despite its potential for both improving resource utilization as well as providing resource guarantees to its users. In this article, we systematically evaluate three major virtual machine implementations for computationally intensive HPC applications using various standard benchmarks. Using VMWare Server, Xen, and OpenVZ, we examine the suitability of full virtualization (VMWare), paravirtualization (Xen), and operating system-level virtualization (OpenVZ) in terms of network utilization, SMP performance, file system performance, and MPI scalability. We show that the operating system-level virtualization provided by OpenVZ provides the best overall performance, particularly for MPI scalability. With the knowledge gained by our VM evaluation, we extend OpenVZ to include support for checkpointing and fault-tolerance for MPI-based virtual server distributed computing.

show abstract

Section: Related Workmentioning

confidence: 57%

A fault-tolerant strategy for virtualized HPC clusters

Walters

Chaudhary

2008

J Supercomput

View full text Add to dashboard Cite

show abstract

“…The remaining work on distributed transparent checkpointing can be divided into two categories: 1) User-level MPI libraries for checkpointing [4], [5], [12], [14], [15], [32], [34], [36], [37]: works for distributed processes, but only if they communicate exclusively through MPI (Message Passing Interface). Typically restricted to a particular dialect of MPI.…”

Section: Related Workmentioning

confidence: 99%

DMTCP: Transparent checkpointing for cluster computations and the desktop

Ansel

Arya

Coopermany

2009

2009 IEEE International Symposium on Parallel &Amp; Distributed Processing

238

185

View full text Add to dashboard Cite

DMTCP (Distributed MultiThreaded CheckPointing) is a transparent user-level checkpointing package for distributed applications. Checkpointing and restart is demonstrated for a wide range of over 20 well known applications, including MATLAB, Python, TightVNC, MPICH2, OpenMPI, and runCMS. RunCMS runs as a 680 MB image in memory that includes 540 dynamic libraries, and is used for the CMS experiment of the Large Hadron Collider at CERN. DMTCP transparently checkpoints general cluster computations consisting of many nodes, processes, and threads; as well as typical desktop applications. On 128 distributed cores (32 nodes), checkpoint and restart times are typically 2 seconds, with negligible run-time overhead. Typical checkpoint times are reduced to 0.2 seconds when using forked checkpointing. Experimental results show that checkpoint time remains nearly constant as the number of nodes increases on a medium-size cluster.DMTCP automatically accounts for fork, exec, ssh, mutexes/semaphores, TCP/IP sockets, UNIX domain sockets, pipes, ptys (pseudo-terminals), terminal modes, ownership of controlling terminals, signal handlers, open file descriptors, shared open file descriptors, I/O (including the readline library), shared memory (via mmap), parent-child process relationships, pid virtualization, and other operating system artifacts. By emphasizing an unprivileged, user-space approach, compatibility is maintained across Linux kernels from 2.6.9 through the current 2.6.28. Since DMTCP is unprivileged and does not require special kernel modules or kernel patches, DMTCP can be incorporated and distributed as a checkpoint-restart module within some larger package.

show abstract

“…Our work is different from theirs in the respect that we are analyzing the benefits of different RDMA semantics to reduce the number of control messages. Majumder et al [18] have proposed an event based progress mechanism for LA-MPI [9]. They indicate the benefits of such an approach to overlap in applications.…”

Section: Related Workmentioning

confidence: 99%

RDMA read based rendezvous protocol for MPI over InfiniBand

Sur

Jin

Chai

et al. 2006

Proceedings of the Eleventh ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming

View full text Add to dashboard Cite

Message Passing Interface (MPI) is a popular parallel programming model for scientific applications. Most high-performance MPI implementations use Rendezvous Protocol for efficient transfer of large messages. This protocol can be designed using either RDMA Write or RDMA Read. Usually, this protocol is implemented using RDMA Write. The RDMA Write based protocol requires a twoway handshake between the sending and receiving processes. On the other hand, to achieve low latency, MPI implementations often provide a polling based progress engine. The two-way handshake requires the polling progress engine to discover multiple control messages. This in turn places a restriction on MPI applications that they should call into the MPI library to make progress. For compute or I/O intensive applications, it is not possible to do so. Thus, most communication progress is made only after the computation or I/O is over. This hampers the computation to communication overlap severely, which can have a detrimental impact on the overall application performance. In this paper, we propose several mechanisms to exploit RDMA Read and selective interrupt based asynchronous progress to provide better computation/communication overlap on InfiniBand clusters. Our evaluations reveal that it is possible to achieve nearly complete computation/communication overlap using our RDMA Read with Interrupt based Protocol. Additionally, our schemes yield around 50% better communication progress rate when computation is overlapped with communication. Further, our application evaluation with Linpack (HPL) and NAS-SP (Class C) reveals that MPI Wait time is reduced by around 30% and 28%, respectively, for a 32 node InfiniBand cluster. We observe that the gains obtained in the MPI Wait time increase as the system size increases. This indicates that our designs have a strong positive impact on scalability of parallel applications.

show abstract

A network-failure-tolerant message-passing system for terascale clusters

Cited by 31 publications

References 2 publications

A fault-tolerant strategy for virtualized HPC clusters

A fault-tolerant strategy for virtualized HPC clusters

DMTCP: Transparent checkpointing for cluster computations and the desktop

RDMA read based rendezvous protocol for MPI over InfiniBand

Contact Info

Product

Resources

About