Breaking the Connection: RDMA Deconstructed

Sivaram, R.; Govindaraju, Rama; Hochschild, Peter; Blackmore, Robert; Chaudhary, Piyush

doi:10.1109/conect.2005.9

Cited by 7 publications

(5 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the performance comparison does not show a significant performance improvement on NAS Parallel Benchmarks [15]. Govindaraju et al presented the design of the IBM High Performance Switch (HPS) Interconnect and a performance evaluation using simple micro-benchmarks [16], [17]. IBM HPS pro-vides Remote Direct Memory Access (RDMA) capability on connectionless transport semantics.…”

Section: Related Workmentioning

confidence: 99%

Evaluating the Potential of Cray Gemini Interconnect for PGAS Communication Runtime Systems

Vishnu

Bruggencate²,

Olson³

2011

2011 IEEE 19th Annual Symposium on High Performance Interconnects

View full text Add to dashboard Cite

The Cray Gemini Interconnect has been recently introduced as the next generation network for building scalable multi-petascale supercomputers. The Cray XE6 systems, which use the Gemini Interconnect are becoming available with Message Passing Interface (MPI) and Partitioned Global Address Space (PGAS) Models such as as Global Arrays, Unified Parallel C, Co-Array Fortran and Cascade High Performance Language. These PGAS models use one-sided communication runtime systems such as MPI-Remote Memory Access, Aggregate Remote Memory Copy Interface and proprietary communication runtime systems. The primary objective of our work is to study the potential of Cray Gemini Interconnect by designing application specific micro-benchmarks using the DMAPP userspace library. We design micro-benchmarks to study the performance of simple communication primitives and application specific microbenchmarks to understand the behavior of Gemini Interconnect at scale. In our experiments, the Gemini Interconnect can achieve a peak bandwidth of 6911 MB/s and a latency of 1µs for get communication primitive. Scalability tests for atomic memory operations and shift communication operation up to 65536 processes show the efficacy of the Gemini Interconnect.

show abstract

Section: Related Workmentioning

confidence: 99%

Evaluating the Potential of Cray Gemini Interconnect for PGAS Communication Runtime Systems

Vishnu

Bruggencate²,

Olson³

2011

2011 IEEE 19th Annual Symposium on High Performance Interconnects

View full text Add to dashboard Cite

show abstract

“…A popular mechanism to alleviate hot-spots in the network is to leverage multi-rail clusters [6,14,18,19]. Coll et al have proposed the general algorithms for utilization of multi-rail clusters and studied their benefits with point-to-point and collective communication primitives [18].…”

Section: Related Studymentioning

confidence: 99%

“…A popular mechanism to alleviate hot-spots in the network is to leverage multi-rail clusters [6,14,18,19]. proposed by Vishnu et al [14,20].…”

Section: Related Studymentioning

confidence: 99%

See 1 more Smart Citation

Topology agnostic hot‐spot avoidance with InfiniBand

Vishnu¹,

Koop²,

Moody

et al. 2008

Concurrency and Computation

View full text Add to dashboard Cite

SUMMARY InfiniBand has become a very popular interconnect due to its advanced features and open standard.Large-scale InfiniBand clusters are becoming very popular, as reflected by the TOP 500 supercomputer rankings. However, even with popular topologies such as constant bi-section bandwidth Fat Tree, hot-spots may occur with InfiniBand due to inappropriate configuration of network paths, presence of other jobs in the network and un-availability of adaptive routing. In this paper, we present a hot-spot avoidance layer (HSAL) for InfiniBand, which provides hot-spot avoidance using path bandwidth estimation and multi-pathing using LMC mechanism, without taking the network topology into account. We propose an adaptive striping policy with batch-based striping and sorting approach, for efficient utilization of disjoint network paths. Integration of HSAL with MPI, the de facto programming model of clusters, shows promising results with collective communication primitives and MPI applications.

show abstract

“…However, our focus is on addressing the scalability issues of one-sided network primitives for clusters comprising of large number of nodes. The benefits of RDMA over UD have been explored in [8]. The authors explore the potential of using RDMA over UD for increased performance, reduction in hot-spot effects and caching problems.…”

Section: Related Workmentioning

confidence: 99%

On using connection-oriented vs. connection-less transport for performance and scalability of collective and one-sided operations

Mamidala

Narravula

Vishnu

et al. 2007

Proceedings of the 12th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming

View full text Add to dashboard Cite

Communication subsystem plays a pivotal role in achieving scalable performance in clusters. The communication semantics employed are dictated by the programming model used by the application such as MPI, UPC, etc. Out of the gamut of communication primitives, collective and one-sided operations are especially significant and have to be designed harnessing the capabilities and features exposed by the underlying networks. In some cases, there is a direct match between the semantics of the operations and the underlying network primitives. InfiniBand provides two transport modes: (i) Connection-oriented Reliable connection (RC) supporting Memory and Channel semantics and (ii) Connection-less Unreliable Datagram (UD) supporting Channel semantics. Achieving good performance and scalability needs careful analysis and design of communication primitives based on these options.In this paper, we evaluate the scalability and performance tradeoffs between RC and UD transport modes. We study the semantic advantages of mapping collective and one-sided operations on to memory and channel semantics of InfiniBand(IBA). We take AlltoAll as a case study to demonstrate the benefits of RDMA over Send/Recv and to show the performance/memory trade-offs over InfiniBand transports. Our experimental results show that UDbased AlltoAll performs 38% better than Bruck's algorithm for short messages and up to two times better than the direct AlltoAll over RC. Since IBA does not provide RDMA over UD in hardware, we emulate the same in our study. Our results show a performance dip of up to a factor of three for emulated RDMA Read latency as compared to RC, highlighting the need for hardware based RDMA operations over UD.

show abstract

Breaking the Connection: RDMA Deconstructed

Cited by 7 publications

References 12 publications

Evaluating the Potential of Cray Gemini Interconnect for PGAS Communication Runtime Systems

Evaluating the Potential of Cray Gemini Interconnect for PGAS Communication Runtime Systems

Topology agnostic hot‐spot avoidance with InfiniBand

On using connection-oriented vs. connection-less transport for performance and scalability of collective and one-sided operations

Contact Info

Product

Resources

About