A Speculative and Adaptive MPI Rendezvous Protocol Over RDMA-enabled Interconnects

Abstract

Overlapping computation with communication is a key technique to conceal the effect of communication latency on the performance of parallel applications. Message Passing Interface (MPI) is a widely used message passing standard for high performance computing. One of the most important factors in achieving a good level of overlap is the MPI ability to make progress on outstanding communication operations. In this paper, we propose a novel speculative MPI Rendezvous protocol that uses RDMA Read and RDMA Write to effectively improve communication progress and consequently the overlap ability. Performance results based on a modified MPICH2 implementation over 10-Gigabit iWARP Ethernet reveal a significant (80-100%) improvement in receiver side overlap and progress ability. We have also observed up to 30% improvement in application wait time for some NPB applications as well as the RADIX application. For applications that do not benefit from this protocol, an adaptation mechanism is used to stop the speculation to effectively reduce the protocol overhead.

Intra-Epoch Message Scheduling To Exploit Unused or Residual Overlapping Potential

“…The literature is particularly abundant for communication/computation overlapping for two-sided communications [8,12,13,19]. In order to mitigate the synchronization effects, previous MPI onesided communication designs deferred the blocking steps to the epoch-closing routine [1,14].…”

Profile Guided MPI Protocol Selection for Point-to-Point Communication Calls

“…Yuan et al refine the traditional rendezvous protocol over RDMA [9] presented by others [10], [7], [8] through two protocols they denote receiver initiated and sender initiated. In the case of receiver initiated rendezvous protocol, the receiver notifies the sender of the user buffer's address.…”

“…Upon arrival, the receiver performs a RDMA read operation on the sender's buffer to transfer the data. A speculative and adaptive approach for MPI rendezvous over RDMA was proposed [8] in which the task that arrives early chooses the communication protocol.…”

An Efficient MPI Message Queue Mechanism for Large-scale Jobs

“…For InfiniBand [2], TupleQ [16] has been proposed where a Shared Receive Queue (SRQ) is created for each <contextId, rank, tag> tuple to match messages in hardware and improve the Rendezvous protocol [18]. Because each created SRQ is never freed, TupleQ is not suitable for large applications, even when there is no queue buildup.…”