Asynchronous zero-copy communication for synchronous sockets in the sockets direct protocol (SDP) over InfiniBand

Abstract

Sockets Direct Protocol (SDP) is an industry standard pseudo socketslike implementation to allow existing sockets applications to directly and transparently take advantage of the advanced features of current generation networks such as InfiniBand. The SDP standard supports two kinds of sockets semantics, viz., Synchronous sockets (e.g., used by Linux, BSD, Windows) and Asynchronous sockets (e.g., used by Windows, upcoming support in Linux). Due to the inherent benefits of asynchronous sockets, the SDP standard allows several intelligent approaches such as source-avail and sink-avail based zero-copy for these sockets. Unfortunately, most of these approaches are not beneficial for the synchronous sockets interface. Further, due to its portability, ease of use and support on a wider set of platforms, the synchronous sockets interface is the one used by most sockets applications today. Thus, a mechanism by which the approaches proposed for asynchronous sockets can be used for synchronous sockets is highly desirable. In this paper, we propose one such mechanism, termed as AZ-SDP (Asynchronous Zero-Copy SDP), where we memory-protect application buffers and carry out communication asynchronously while maintaining the synchronous sockets semantics. We present our detailed design in this paper and evaluate the stack with an extensive set of benchmarks. The experimental results demonstrate that our approach can provide an improvement of close to 35% for medium-message unidirectional throughput and up to a factor of 2 benefit for computationcommunication overlap tests and multi-connection benchmarks.

Evaluation of ConnectX Virtual Protocol Interconnect for Data Centers

“…The authors found that improvements could be made to throughput for larger message sizes. We subsequently introduced an asynchronous zero-copy mechanism [1] providing better performance than that obtained by Goldenberg et al…”

Designing next generation data-centers with advanced communication protocols and systems services

“…Due to the inherent benefits of asynchronous sockets, the SDP standard also allows several intelligent approaches such as source-avail and sink-avail based zero-copy for these sockets. However, most of these approaches that work well for the asynchronous sockets interface are not as beneficial for the synchronous sockets interface [4]. Further, due to its portability, ease of use and support on a wider set of platforms, the synchronous sockets interface is the one used by most sockets applications today.…”

“…In our approach, the key idea in meeting these design goals is to memory-protect the user buffer (thus disallow the application from accessing it) and to carry out communication asynchronously from this buffer, while tricking the application into believing that we are carrying out data communication in a synchronous manner. More details about the design of the AZ-SDP scheme can be found in [4].…”

“…Also, for medium messages, AZ-SDP performs the best with about 35% improvement compared to ZSDP. More microbenchmark results such as communication and computation overlap and impact of page faults on the performance of AZ-SDP can be found in [4].…”

Boosting Ethernet Performance by Segment-Based Routing

“…Zero copy protocols [3,27] and remote memory access techniques [19] have been added to Ethernet interfaces in order to improve latency. Also, the addition of an on/off based flow-control mechanism to the Ethernet standard in 1997 [24], allows Ethernet to behave as a loss-less network [21] without the need for expensive retransmission and acknowledgement of packets.…”