Improving Communication Progress and Overlap in MPI Rendezvous Protocol over RDMA-enabled Interconnects

Abstract

Overlapping computation with communication is a key technique to conceal the effect of communication latency on the performance of parallel applications. MPI is a widely used message passing standard for high performance computing. One of the most important factors in achieving a good level of overlap is the MPI ability to make progress on outstanding communication operations.In this paper, we address some of the communication progress shortcomings in the current polling and RDMA Read based Rendezvous protocol used for transferring large messages in MPI. We then propose a novel speculative Rendezvous protocol that uses RDMA Read and RDMA Write to effectively improve communication progress and consequently the overlap ability. Performance results based on a modified MPICH2 over 10-Gigabit iWARP Ethernet reveal a significant (80-100%) improvement in receiver side overlap and progress ability.

Maximizing MPI point-to-point communication performance on RDMA-enabled clusters with customized protocols

“…However, such a combination does not yield ideal performance in many situations. In particular, the senderinitiated rendezvous protocol introduces unnecessary synchronizations between the sender and the receiver, requires more rounds of messaging than needed in common situations, and has a limited capability for supporting communication and computation overlaps [2,15,16,18]. These limitations may introduce significant performance penalties in MPI applications.…”

“…Moreover, simultaneously supporting the sender-initiated eager/rendezvous protocols and the receiver-initiated rendezvous protocol without introducing excessive overheads posts significant challenges. Existing proposals [2,15,16] fall short of fully addressing these challenges. The Gravel library [2] provides mechanisms to support both the sender-initiated and receiver-initiated protocols.…”

“…In particular, receiverinitiated rendezvous schemes have been proposed to improve the communication performance of the sender-initiated rendezvous protocol [2,15,16]. Although the receiver-initiated protocol can in principle overcome some problems in the sender-initiated protocol, the receiver-initiated protocol has its own limitations.…”

“…This approach cannot support full MPI functionality such as MPI ANY SOURCE [15]. The scheme in [16] supports both sender-initiated and receiver-initiated rendezvous protocols simultaneously. However, among other issues, this scheme requires an acknowledgment message for each control message, which introduces significant overheads.…”

Profile Guided MPI Protocol Selection for Point-to-Point Communication Calls

“…Yuan et al refine the traditional rendezvous protocol over RDMA [9] presented by others [10], [7], [8] through two protocols they denote receiver initiated and sender initiated. In the case of receiver initiated rendezvous protocol, the receiver notifies the sender of the user buffer's address.…”