ConnectX-2 CORE-Direct Enabled Asynchronous Broadcast Collective Communications

Venkata, Manjunath Gorentla; Graham, Richard L.; Ladd, Joshua; Shamis, Pavel; Rabinovitz, Ishai; Filipov, Vasily; Shainer, Gilad

doi:10.1109/ipdps.2011.221

Cited by 15 publications

(6 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As a principled approach to network offloading, sPIN has the potential to replace specific offload solutions such as ConnectX CORE-Direct collective offload [10], Cray Aries [2], IBM PERCS [11], or Portals 4 [12] triggered operations. Instead, the community can focus on developing domain or application-specific sPIN libraries to accelerate networking, very much like NVIDIA's cuBLAS or Vien-naCL [13].…”

Section: Motivationmentioning

confidence: 99%

“…All benefits known from collective offloading implementations [10,12,29] such as asynchronous progression and noiseresilience remain true for sPIN. As opposed to existing offloading frameworks that restrict the collective algorithms (e.g., to pre-defined trees), sPIN supports arbitrary algorithms (including pipeline and double-tree [30]) due to the flexible programmability and high forwarding performance of the HPUs.…”

Section: Spin Offloaded Broadcastmentioning

confidence: 99%

See 1 more Smart Citation

sPIN

Hoefler

Girolamo

Taranov

et al. 2017

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

Optimizing communication performance is imperative for largescale computing because communication overheads limit the strong scalability of parallel applications. Today's network cards contain rather powerful processors optimized for data movement. However, these devices are limited to fixed functions, such as remote direct memory access. We develop sPIN, a portable programming model to offload simple packet processing functions to the network card. To demonstrate the potential of the model, we design a cycle-accurate simulation environment by combining the network simulator Log-GOPSim and the CPU simulator gem5. We implement offloaded message matching, datatype processing, and collective communications and demonstrate transparent full-application speedups. Furthermore, we show how sPIN can be used to accelerate redundant in-memory filesystems and several other use cases. Our work investigates a portable packet-processing network acceleration model similar to compute acceleration with CUDA or OpenCL. We show how such network acceleration enables an eco-system that can significantly speed up applications and system services.ACM Reference Format:

show abstract

Section: Motivationmentioning

confidence: 99%

Section: Spin Offloaded Broadcastmentioning

confidence: 99%

sPIN

Hoefler

Girolamo

Taranov

et al. 2017

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

show abstract

“…This management queue allows to delay certain operations until others are finished, and therefore to express dependencies between operations. Researchers reported positive results when implementing single collectives such as barrier and broadcast with this technology [26], [27]. To the best of our knowledge no one has attempted to show that the primitives offered by CORE-Direct are powerful enough to offload any communication schedule, or shown its limits.…”

Section: Experimental Evaluationmentioning

confidence: 99%

Protocols for Fully Offloaded Collective Operations on Accelerated Network Adapters

Schneider

Hoefler

Grant

et al. 2013

2013 42nd International Conference on Parallel Processing

View full text Add to dashboard Cite

Abstract-With each successive generation, network adapters for high-performance networks are becoming more powerful and feature rich. High-performance NICs can now provide support for performing complex group communication operations on the NIC without any host CPU involvement. Several "offloading interfaces" have been designed with the collective communications goal being the complete offloading of arbitrary communication patterns.In this work, we analyze the offloading model offered in the Portals 4 specification in detail. We perform a theoretical analysis based on abstract communication graphs and show several protocols for implementing offloaded communication schedules. Based on our analysis, we propose and implement an extension to the portals 4 specification that enables offloading any communication pattern completely to the NIC. Our measurements with several advanced communication algorithms confirm that the enhancements provide good overlap and asynchronous progress in practical settings. Altogether, we demonstrate a complete and simple scheme for implementing arbitrary offloaded communication algorithms and hardware. Our protocols can act as a blueprint for the development of communication hardware as well as middleware while optimizing the whole communication stack. I. MOTIVATIONMoore's law is still going strong despite the end of frequency and Dennard scaling. CPU and chip vendors have managed to maintain Moore's law by going broad, i.e., duplicate functional units (e.g., cores or vector units) and/or add new functionality to chips. Thus, several microprocessor vendors began extending core functionalities of chips. For example, functionalities that were traditionally in a north bridge, such as a memory controller, are now commonly included in main CPUs. Similarly, networking chips have become more powerful and are to be integrated into next-generation CPUs.The growing number of cores per network endpoint increases the requirements for the network and memory interfaces. Modern multi-core CPUs already scale the number of memory controllers with the cores, similarly, network interfaces may follow. High-performance networks provide much more complex functionality than memory controllers. Thus, it seems reasonable to devote some silicon to performing advanced functions. The most important parameters of today's networks are latency, bandwidth, and message-rate. Therefore, modern networks are highly tuned to provide high performance for these metrics. However, many algorithms from scientific computing and other fields, such as databases, operating systems, and financial computations, use advanced communication algorithms over sets of processes. These are often called "collective communications" in high-performance computing (HPC) and they are important to many types of applications. Their increased complexity over standard point-to-point communications and participation of multiple processes make them important to overall communication performance, as well as a prime target for efforts to enhance network pe...

show abstract

“…A big motivation for this work, as well as our previous collective algorithm work [16], is to develop collective communications that provide a high-degree of ability to overlap computation and communication leading to effective system utilization. While providing the ability to overlap and providing high-performance communication is essential, it is just as important to provide the basic capabilities needed to implement asynchronous communication algorithms.…”

Section: B Communication-computation Overlapmentioning

confidence: 99%

Exploring the All-to-All Collective Optimization Space with ConnectX CORE-Direct

Venkata

Graham

Ladd

et al. 2012

2012 41st International Conference on Parallel Processing

Self Cite

View full text Add to dashboard Cite

The all-to-all collective communication operation is used by many scientific applications, and is one of the most time consuming and challenging collective operation to optimize. The algorithms for all-to-all operations typically fall into two classes, logarithmic and linear scaling algorithms, with Bruck's algorithm, a logarithmic scaling algorithm, used in many smalldata all-to-all implementations. The recent addition of InfiniBand CORE-Direct support for network management of collective communications offers new opportunities for optimizing all-toall operation as well as supporting truly asynchronous implementations of these operations. This paper presents several new enhancements to the Bruck small-data algorithm that leverage CORE-Direct and other InfiniBand network capabilities to produce efficient implementations of this collective operation. These include RDMA, SR-RNR, and SR-RTR algorithms. In addition, nonblocking implementations of these collective operations are also presented.Benchmark results show that the RDMA algorithm, which uses CORE-Direct capabilities to offload collective communication management to the Host Channel Adapter (HCA), hardware gather support for sending non-continuous data, and low-latency RDMA semantics, performs the best. For a 64 processes and 128 byte-per-process all-to-all, the RDMA algorithm performs 27% better than Bruck's algorithm implementation in Open MPI and 136% better than the SR-RTR algorithm. In addition, the nonblocking versions of these algorithms have the same performance characteristics as the blocking algorithms. Finally, measurements of computation/communication overlap capacity show that all offloaded algorithms achieve about 98% overlap for large data all-to-all, whereas implementations using host-based progress achieve only about 9.5% overlap.

show abstract

ConnectX-2 CORE-Direct Enabled Asynchronous Broadcast Collective Communications

Cited by 15 publications

References 8 publications

sPIN

sPIN

Protocols for Fully Offloaded Collective Operations on Accelerated Network Adapters

Exploring the All-to-All Collective Optimization Space with ConnectX CORE-Direct

Contact Info

Product

Resources

About