RDMA over Ethernet &amp;#x2014; A preliminary study

Subramoni, Hari; Lai, Ping; Luo, Miao; Panda, Dhabaleswar K.

doi:10.1109/clustr.2009.5289144

Cited by 31 publications

(12 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Open-MX allows for Myrinet Express networking traffic to function over Ethernet hardware (MXoE) [12]. Like the local-area network (LAN) RDMA technology in MXoE, RDMA over Converged Ethernet (RoCE) [47] has been proposed as an alternative RDMA over Ethernet technology. RoCE uses InfiniBand's RDMA stack directly, using Ethernet as the link/physical layer.…”

Section: Introductionmentioning

confidence: 99%

Scalable connectionless RDMA over unreliable datagrams

et al. 2015

View full text Add to dashboard Cite

a b s t r a c tThe overhead imposed by connection-based protocols for high-performance computing (HPC) systems can be detrimental to system resource usage and performance. This paper demonstrates for the first time a unified send/recv and Remote Direct Memory Access (RDMA) Write over datagrams design for RDMA-capable network adapters. We previously designed the first and only unreliable datagram RDMA model, RDMA Write-Record, and demonstrated its superior performance over connection-based RDMA. RDMA Write-Record can be applied to several RDMA capable networks, such as iWARP and InfiniBand (which does not support unreliable RDMA Writes). iWARP is a state-of-the-art, high-speed, connection-based RDMA networking technology for both local and wide-area Ethernet networks. iWARP is used as the platform to demonstrate our unreliable RDMA operation design for both channel and memory semantics. We previously outlined the requirements for extending iWARP to operate over datagrams.Here we extend our work on commercial datacenter applications by providing broadcast support for send/recv. In order to study the scalability of datagram-iWARP, we added Message Passing Interface support for RDMA Write-Record to investigate the scalability of HPC-based scientific applications for both send/recv and RDMA Write-Record. The results show that both models outperform their connection-based alternatives, providing superior performance and scalability in a software prototype.

show abstract

Section: Introductionmentioning

confidence: 99%

Scalable connectionless RDMA over unreliable datagrams

et al. 2015

View full text Add to dashboard Cite

show abstract

“…As IPoIB is the interface by which socket based programs operate over IB, it is important to examine its performance in current data center environments to understand the benefits that might arise from future hybrid data center networking environments. Convergence efforts such as RDMAoE/IBoE [24], RDMAoCEE [5] and VPI [7] are already underway to combine existing high performance networks with Ethernet. A new set of standards referred to as Converged Enhanced Ethernet (CEE) [11][12][13][14][15]17] has opened up issues revolving around providing advanced features over Ethernet networks.…”

Section: Introductionmentioning

confidence: 99%

“…Some industry vendors and researchers [5] are also proposing to include RDMA functionality over CEE [5]. Technologies such as InfiniBand over Ethernet (IBoE) [24] provide for InfiniBand packets to be encapsulated into an Ethernet frame, while still utilizing the IB network and transport layers. This allows for the features of InfiniBand to be provided over an Ethernet network, including RDMA.…”

Section: Introductionmentioning

confidence: 99%

A study of hardware assisted IP over InfiniBand and its impact on enterprise data center performance

Grant

Balaji

Afsahi

2010

2010 IEEE International Symposium on Performance Analysis of Systems &Amp; Software (ISPASS)

View full text Add to dashboard Cite

Abstract-High-performance sockets implementations such as the Sockets Direct Protocol (SDP) have traditionally showed major performance advantages compared to the TCP/IP stack over InfiniBand (IPoIB). These stacks bypass the kernel-based TCP/IP and take advantage of network hardware features, providing enhanced performance. SDP has excellent performance but limited utility as only applications relying on the TCP/IP sockets API can use it and other IP stack uses (IPSec, UDP, SCTP) or TCP layer modifications (iSCSI) cannot benefit from it. Recently, newer generations of InfiniBand adapters, such as ConnectX from Mellanox, have provided hardware support for the IP stack itself, such as Large Send Offload and Large Receive Offload. As such high performance socket networks are likely to be deployed or converged with existing Ethernet networking solutions, the performance of such technologies is important to assess. In this paper we take a first look at the performance advantages provided by these offload techniques and compare them to SDP. Our micro-benchmarks and enterprise data-center experiments show that hardware assisted IPoIB can provide competitive performance with SDP and even outperform it in some cases.

show abstract

“…Subramoni [20] also presented another driver to the GridFTP framework to merge the capability of InfiniBand and GridFTP. RDMA technology was extended and integrated into Message Passing Interface (MPI) [21], [14], [22] to allow parallel applications to take advantage of RDMA's low latency and high performance communication capability. However, its scalability and performance are not yet tested and validated in the newly available 40Gbps InfiniBand and Ethernet network environments.…”

Section: Related Workmentioning

confidence: 99%

Protocols for wide-area data-intensive applications: Design and performance issues

Ren¹,

Li²,

Yu³

et al. 2012

2012 International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

Abstract-Providing high-speed data transfer is vital to various data-intensive applications. While there have been remarkable technology advances to provide ultra-high-speed network bandwidth, existing protocols and applications may not be able to fully utilize the bare-metal bandwidth due to their inefficient design. We identify the same problem remains in the field of Remote Direct Memory Access (RDMA) networks. RDMA offloads TCP/IP protocols to hardware devices. However, its benefits have not been fully exploited due to the lack of efficient software and application protocols, in particular in wide-area networks. In this paper, we address the design choices to develop such protocols. We describe a protocol implemented as part of a communication middleware. The protocol has its flow control, connection management, and task synchronization. It maximizes the parallelism of RDMA operations. We demonstrate its performance benefit on various local and wide-area testbeds, including the DOE ANI testbed with RoCE links and InfiniBand links.

show abstract

RDMA over Ethernet — A preliminary study

Cited by 31 publications

References 3 publications

Scalable connectionless RDMA over unreliable datagrams

Scalable connectionless RDMA over unreliable datagrams

A study of hardware assisted IP over InfiniBand and its impact on enterprise data center performance

Protocols for wide-area data-intensive applications: Design and performance issues

Contact Info

Product

Resources

About