Maximizing MPI point-to-point communication performance on RDMA-enabled clusters with customized protocols

Abstract

Message Passing Interface (MPI) point-to-point communications are usually realized with two protocols, the eager protocol for small messages and the rendezvous protocol for medium and large sized messages. Traditional senderinitiated rendezvous protocols are sub-optimal in many situations. In this work, we propose to refine the rendezvous protocol for medium and large messages on RDMA-enabled clusters with three protocols that are customized for different situations, a hybrid protocol for medium sized messages when the sender arrives early, a sender-initiated protocol for large messages when the sender arrives early, and a receiver-initiated protocol when the receiver arrives early. In comparison to traditional sender-initiated rendezvous protocols, the proposed scheme reduces unnecessary synchronizations, decreases the number of control messages that are in the critical path of communications, and improves the communication progress, which results in a significantly better communication-computation overlap capability. We present and analyze these protocols, and describe how these protocols and the eager protocol can be seamlessly integrated in one system without introducing an excessive number of control messages. We have implemented the proposed scheme for InfiniBand clusters. The experimental results demonstrate the effectiveness of the proposed technique.

Intra-Epoch Message Scheduling To Exploit Unused or Residual Overlapping Potential

“…The literature is particularly abundant for communication/computation overlapping for two-sided communications [8,12,13,19]. In order to mitigate the synchronization effects, previous MPI onesided communication designs deferred the blocking steps to the epoch-closing routine [1,14].…”

Profile Guided MPI Protocol Selection for Point-to-Point Communication Calls

“…Recent work [9] introduces a new protocol for point-topoint communication over RDMA for certain message sizes. The protocol decouples sender from receiver by making a local copy of the user buffer and notifies the receiver about the address of the buffer.…”

“…Yuan et al refine the traditional rendezvous protocol over RDMA [9] presented by others [10], [7], [8] through two protocols they denote receiver initiated and sender initiated. In the case of receiver initiated rendezvous protocol, the receiver notifies the sender of the user buffer's address.…”

“…First, we discuss protocols we consider, including our pre-copy protocol [9]. Then we discuss the novel parts of our system, including a new sender-initiated post-copy protocol (the dual of our pre-copy protocol), our protocol execution models, and our online profile-based protocol decision system.…”

“…Pre-Copy Protocol: The sender-side pre-copy protocol presented by Yuan et al [9] de-couples the sender from the receiver by making a local copy of the incoming buffer. The local buffer is RDMA-registered during MPI_Init to save the registration cost for every send call.…”