StaR: Breaking the Scalability Limit for RDMA

Wang, Xizheng; Chen, Guo; Yin, Xi-Jun; Dai, Huichen; Li, Bojie; Fu, Binzhang; Tan, Kun

doi:10.1109/icnp52444.2021.9651935

Cited by 15 publications

(7 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…RC is the most widely-used RDMA transport mode, but it is far from perfection because offloading the whole transport process from the host to the NIC has two drawbacks. The first is the scalability problem which is already discussed in previous literature [5]. The second is the availability problem which we discuss in this paper.…”

Section: Insightsmentioning

confidence: 79%

“…The main reason is that RDMA implements a sophisticated network stack in the hardware, making it difficult to restore the NIC correctly and quickly in the cases of NIC hot upgrade and network failures in the NIC. The on-NIC RDMA engine maintains three kinds of states, all stored and accessed on the NIC: DMA-related, security-related, and networking-related [5]. For connectionoriented transport modes (i.e., RC and UC), many contexts reside on the RNIC, such as the source and destination QP numbers, the SQ and RQ states, and the transmission and retransmission states.…”

Section: B Rdma Lacks Of Ha In Cloud Data Centermentioning

confidence: 99%

“…Current RDMA-based applications are still lacking in HA. On the one hand, RDMA is implemented as a stateful transport whose states are mostly maintained on the NIC [5]. Maintaining the strong consistency of the transport's states between the host and the NIC is challenging, which is demanded in 978-1-6654-8018-5/22/$31.00 © 2022 IEEE the cases of NIC hot upgrade and fault recovery.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

DTS: A Dual Transport Switching Scheme for RDMA-based Applications

Chen

Wang

et al. 2022

2022 IEEE International Performance, Computing, and Communications Conference (IPCCC)

View full text Add to dashboard Cite

RDMA has been widely adopted by distributed applications for its low latency, high throughput, and nearzero CPU overhead. However, high availability is still lacked for RDMA-based applications, especially in the cases of network failure or NIC upgrade. In this paper, we propose DTS, a scheme for RDMA-based applications to provide high availability by adopting the active and standby transports and switching them transparently and fast when necessary. We implement the prototype of DTS based on the Libfabric RxD provider. Our evaluation demonstrates that with DTS existing RDMAbased applications can be deployed with zero downtime and the performance degradation is ignorable.

show abstract

Section: Insightsmentioning

confidence: 79%

Section: B Rdma Lacks Of Ha In Cloud Data Centermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

DTS: A Dual Transport Switching Scheme for RDMA-based Applications

Chen

Wang

et al. 2022

2022 IEEE International Performance, Computing, and Communications Conference (IPCCC)

View full text Add to dashboard Cite

show abstract

“…Hardware Solutions: Various studies have explored hardware solutions, such as StaR [27], which deploys a novel receiving module on RDMA NICs. StaR employs an asymmetric state maintenance strategy to separate connection and retransmission states, thereby enhancing QP scalability.…”

Section: Related Workmentioning

confidence: 99%

An Optimized RDMA QP Communication Mechanism for Hyperscale AI Infrastructure

Wang,

Lin,

Zhang

et al. 2024

Preprint

View full text Add to dashboard Cite

The current artificial intelligence (AI) infrastructure widely employs remote direct memory access (RDMA) protocol for high-performance communication in networks, utilizing Reliable Connection (RC)-based Queue Pairs (QP) to ensure end-to-end correct and ordered data transmission. However, as the scale of AI infrastructure continues to expand, this RC-based QP communication mechanism faces deficiencies in scalability and is prone to congestion, resulting in degraded network transfer performance. In this paper, we propose an optimized RDMA QP communication mechanism to address scalability and congestion issues in hyper-scale AI infrastructure networks. Firstly, we replace RC-based QPs with Reliable Datagram (RD)-based QPs and propose a new reliable mechanism to address scalability problems, eliminating the need for repetitive QP establishment by AI processes during external communication. Additionally, to mitigate congestion caused by a single path, we implement multipath data transmission by introducing a new unordered reception method in the network software stack. Through experiments and simulation tests, the optimized RDMA QP communication in large-scale AI infrastructure exhibits exceptional scalability and significantly reduces the occurrence of congestion, resulting in an overall network performance improvement of over 15%.

show abstract

“…While RDMA provides protection via rkeys (i.e., a node can access a remote region only if it has the respective rkey), this offers limited authentication capabilities for client requests. For example, having a single rkey for all files would not prevent clients from accessing data they do not own, while having a rkey for each file would require registering #f iles × #rights memory regions, which can lead to scalability issues [30]. On the other side, validating requests on the CPU of the storage nodes increases overhead, requiring either to buffer the full write before committing it to the storage target (losing the zero-copy benefit of RDMA) or to introduce an additional network round-trip before the RDMAwrite.…”

Section: Client Request Authenticationmentioning

confidence: 99%

Building Blocks for Network-Accelerated Distributed File Systems

Girolamo¹,

Sensi²,

Taranov³

et al. 2022

Preprint

View full text Add to dashboard Cite

High-performance clusters and datacenters pose increasingly demanding requirements on storage systems. If these systems do not operate at scale, applications are doomed to become I/O bound and waste compute cycles. To accelerate the data path to remote storage nodes, remote direct memory access (RDMA) has been embraced by storage systems to let data flow from the network to storage targets, reducing overall latency and CPU utilization. Yet, this approach still involves CPUs on the data path to enforce storage policies such as authentication, replication, and erasure coding. We show how storage policies can be offloaded to fully programmable SmartNICs, without involving host CPUs. By using PsPIN, an open-hardware SmartNIC, we show latency improvements for writes (up to 2x), data replication (up to 2x), and erasure coding (up to 2x), when compared to respective CPU-and RDMA-based alternatives.

show abstract

StaR: Breaking the Scalability Limit for RDMA

Cited by 15 publications

References 14 publications

DTS: A Dual Transport Switching Scheme for RDMA-based Applications

DTS: A Dual Transport Switching Scheme for RDMA-based Applications

An Optimized RDMA QP Communication Mechanism for Hyperscale AI Infrastructure

Building Blocks for Network-Accelerated Distributed File Systems

Contact Info

Product

Resources

About