Scalable parallel application launch on Cplant™

Brightwell, Ron; Fisk, Lee Ann

doi:10.1145/582034.582074

Cited by 27 publications

(23 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Brightwell et al [8] present the components of the runtime system for parallel application launch on Cplant project. They do not assume that the executable to be launched is available on a global file system.…”

Section: Related Workmentioning

confidence: 99%

A multi-level scalable startup for parallel applications

Gupta

Zheng

Kalé

2011

Proceedings of the 1st International Workshop on Runtime and Operating Systems for Supercomputers

View full text Add to dashboard Cite

High performance parallel machines with hundreds of thousands of processors and petascale performance are already in use, and even larger Exaflops scale computing systems which may have hundreds of millions of cores are planned. To run parallel applications on machines of such massive scale, one of the biggest challenges is the parallel startup process. This task involves two components: (1) parallel launching of appropriate processes on the given set of processors and (2) setting up communication channels to enable the processes to communicate with each other after process launching has completed. Most current startup mechanisms focus on either using daemons which waste system resources or using a startup manager which becomes a scalability bottleneck. In this paper, we investigate the design and scalability of a SMP-aware, multi-level startup scheme with batching of remote shell sessions, which provides a complete solution to startup of a parallel application and facilitates its management during execution. It monitors process health and can be used to support recovery from failures and provide scalable interaction with the application. We demonstrate the performance and scalability of this scheme by applying it to startup Charm++ applications. In particular, starting up a Charm++ program on 16,384 cores of Ranger (at TACC) with Ethernet as the underlying communication layer takes only 25 seconds and attains a speedup of over 400% compared to MPICH2 startup (using hydra as process manager) and over 800% compared to Open MPI startup on Ranger.

show abstract

Section: Related Workmentioning

confidence: 99%

A multi-level scalable startup for parallel applications

Gupta

Zheng

Kalé

2011

Proceedings of the 1st International Workshop on Runtime and Operating Systems for Supercomputers

View full text Add to dashboard Cite

show abstract

“…MPD (Butler et al 2003) uses a group of daemons, arranged in a ring topology, to scalably start MPICH2 MPI processes. Yod (Brightwell and Fisk 2001) provides similar capabilities in the Cplant software stack.…”

Section: Related Workmentioning

confidence: 99%

A Composition Environment for MPI Programs

Desai

Lusk

Bradshaw

2007

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

While MPI is the most common mechanism for expressing parallelism, MPI programs are not composable by using current MPI process managers or parallel shells. We introduce MPISH2, an MPI process manager analogous to serial Unix shells. It allows the composition of MPI and serial Unix utilities with one another to perform scalable tasks across large numbers of Unix clients. This paper discusses in detail issues of process management and parallel tool composition.

show abstract

“…[1], [2], [3], [4], [5], [6] For many important applications, the quantity of data to be processed exceeds the storage capacity or reasonable transfer times that can be achieved on conventional systems. For these large-data applications, a variety of techniques such as disk aggregation [7], [8], [9], [10], parallel network transfer [11], [12], peer-to-peer distribution [13], and multi-level storage management [2] are essential for constructing usable grid computing systems.…”

Section: Introductionmentioning

confidence: 99%

Efficient access to many small files in a filesystem for grid computing

Thain

Moretti

2007

2007 8th IEEE/ACM International Conference on Grid Computing

View full text Add to dashboard Cite

Abstract-Many potential users of grid computing systems have a need to manage large numbers of small files. However, computing and storage grids are generally optimized for the management of large files. As a result, users with small files achieve performance several orders of magnitude worse than possible. Archival tools and custom storage structures can be used to improve small-file performance, but this requires the end user to change the behavior of the application, which is not always practical. To address this problem, we augment the protocol of the Chirp filesystem for grid computing to improve small file performance. We describe in detail how this protocol compares to FTP and NFS, which are widely used in similar situations. In addition, we observe that changes to the system call interface are necessary to invoke the protocol properly. We demonstrate an order-of-magnitude performance improvement over existing protocols for copying files and manipulating large directory trees.

show abstract

Scalable parallel application launch on Cplant™

Cited by 27 publications

References 8 publications

A multi-level scalable startup for parallel applications

A multi-level scalable startup for parallel applications

A Composition Environment for MPI Programs

Efficient access to many small files in a filesystem for grid computing

Contact Info

Product

Resources

About