RDMA-Based Job Migration Framework for MPI over InfiniBand

Ouyang, Xiangyong; Marcarelli, Sonya; Rajachandrasekar, Raghunath; Panda, Dhabaleswar K.

doi:10.1109/cluster.2010.20

Cited by 21 publications

(11 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As opposed to evacuation in the migratable-objects model, task migration in CoCheck was synchronous with the execution of the system, i.e., before an MPI rank could be migrated, CoCheck had to make sure all communicating ranks would hold back messages until the rank was at its new location. Similar tools have recently used process-level live migration in MPI applications [36], [37], [38]. Those tools combine health monitoring of nodes with live migration to provide proactive fault tolerance.…”

Section: Related Workmentioning

confidence: 99%

Using Migratable Objects to Enhance Fault Tolerance Schemes in Supercomputers

Meneses

Zheng

et al. 2015

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Abstract-Supercomputers have seen an exponential increase in their size in the last two decades. Such a high growth rate is expected to take us to exascale in the timeframe 2018-2022. But, to bring a productive exascale environment about, it is necessary to focus on several key challenges. One of those challenges is fault tolerance. Machines at extreme scale will experience frequent failures and will require the system to avoid or overcome those failures. Various techniques have recently been developed to tolerate failures. The impact of these techniques and their scalability can be substantially enhanced by a parallel programming model called migratable objects. In this paper, we demonstrate how the migratable-objects model facilitates and improves several fault tolerance approaches. Our experimental results on thousands of cores suggest fault tolerance schemes based on migratable objects have low performance overhead and high scalability. Additionally, we present a performance model that predicts a significant benefit of using migratable objects to provide fault tolerance at extreme scale.

show abstract

Section: Related Workmentioning

confidence: 99%

Using Migratable Objects to Enhance Fault Tolerance Schemes in Supercomputers

Meneses

Zheng

et al. 2015

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

show abstract

“…In [8], we have made an initial attempt to optimize (1) and (2) by leveraging RDMA to transfer checkpoint data. However it hasn't totally solved the problem since the heavy IO overhead at (3) still dominates.…”

Section: Introductionmentioning

confidence: 99%

“…Job/process migration [6][7][8][9], a pro-active faulttolerance mechanism, has been proposed as a complement to C/R. During a migration, the processes running on a source node are checkpointed and the checkpoint data is transferred to a healthy spare node where the processes are restarted.…”

Section: Introductionmentioning

confidence: 99%

High Performance Pipelined Process Migration with RDMA

Ouyang

Rajachandrasekar

Besseron

et al. 2011

2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

Self Cite

View full text Add to dashboard Cite

Coordinated Checkpoint/Restart (C/R) is a widely deployed strategy to achieve fault-tolerance. However, C/R by itself is not capable enough to meet the demands of upcoming exascale systems, due to its heavy I/O overhead. Process migration has already been proposed in literature as a pro-active fault-tolerance mechanism to complement C/R. Several popular MPI implementations have provided support for process migration, including MVAPICH2 and OpenMPI. But these existing solutions cannot yield a satisfactory performance.In this paper we conduct extensive profiling on several process migration mechanisms, and reveal that inefficient I/O and network transfer are the principal factors responsible for the high overhead. We then propose a new approach, Pipelined Process Migration with RDMA (PPMR), to overcome these overheads. Our new protocol fully pipelines data writing, data transfer, and data read operations during different phases of a migration cycle. PPMR aggregates data writes on the migration source node and transfers data to the target node via high throughput RDMA transport. It implements an efficient process restart mechanism at the target node to restart processes from the RDMA data streams. We have implemented this Pipelined Process Migration protocol in MVAPICH2 and studied the performance benefits. Experimental results show that PPMR achieves a 10.7X speedup to complete a process migration over the conventional approach at a moderate (8MB) memory usage. Process migration overhead on the application is significantly minimized from 38% to 5% by PPMR when three migrations are performed in succession.

show abstract

“…We have proposed a file-based migration design [102] that creates checkpoint files of the processes on the migration source node and move the files via different transports to the target node to restart the processes. The team has made significant enhancements in data transmission at [101] to directly pump process image through a RDMA data pipeline from migration source node to the target node without any file system IO overhead.…”

Section: End-to-end Reliable Data Transmission In Mvapichmentioning

confidence: 99%

Coordinated Fault Tolerance for High-Performance Computing

Dongarra¹,

Bosilca²

2013

View full text Add to dashboard Cite

RDMA-Based Job Migration Framework for MPI over InfiniBand

Cited by 21 publications

References 22 publications

Using Migratable Objects to Enhance Fault Tolerance Schemes in Supercomputers

Using Migratable Objects to Enhance Fault Tolerance Schemes in Supercomputers

High Performance Pipelined Process Migration with RDMA

Coordinated Fault Tolerance for High-Performance Computing

Contact Info

Product

Resources

About