ConnectX-2 InfiniBand Management Queues: First Investigation of the New Support for Network Offloaded Collective Operations

Graham, Richard L.; Poole, Steve; Shamis, Pavel; Bloch, Gil; Bloch, Noam; Chapman, H.; Kagan, Michael; Shahar, Ariel; Rabinovitz, Ishai; Shainer, Gilad

doi:10.1109/ccgrid.2010.9

Cited by 35 publications

(17 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this figure and all subsequent figures MQ is used to label results obtained using HCA offloading approaches, and PTP is used to label results obtained with point-to-point based collective algorithms progressed by the CPU in main memory. While results for the barrier algorithm have been presented previously [19], the current results are significantly improved at the larger process counts. This isa result of the reduce number of queue pairs used in the algorithm, decreasing the number of network context resources the HCA needs to manage.…”

Section: B Discussionmentioning

confidence: 64%

See 1 more Smart Citation

Overlapping computation and communication: Barrier algorithms and ConnectX-2 CORE-Direct capabilities

Graham

Poole

Shamis

et al. 2010

2010 IEEE International Symposium on Parallel &Amp; Distributed Processing, Workshops and PHD Forum (IPDPSW)

Self Cite

View full text Add to dashboard Cite

Abstract-This paper explores the computation and communication overlap capabilities enabled by the new CORE-Direct hardware capabilities introduced in the InfiniBand (IB) Host Channel Adapter (HCA) ConnectX-2. These capabilities enable the progression and completion of data-dependent communications sequences to progress and complete at the network level without any Central Processing Unit (CPU) involvement. We use the latency dominated nonblocking barrier algorithm in this study, and find that at 64 process count, a contiguous time slot of about 80 percent of the nonblocking barrier time is available for computation. This time slot increases as the number of processes participating increases. In contrast, CPU based implementations provide a time slot of up to 30 percent of the nonblocking barrier time. This bodes well for the scalability of simulations employing offloaded collective operations. These capabilities can be used to reduce the effects of system noise, and when using nonblocking collective operations have the potential to hide the effects of application load imbalance.

show abstract

Section: B Discussionmentioning

confidence: 64%

“…A detailed description of the CORE-Direct support and how this is used to implement support for MPI collective operations in Open MPI [18] is described elsewhere [19]. In this section we provide a brief description of these, as well as very recent enhancements to the MPI support.…”

Section: An Overview Of Infinibandmentioning

confidence: 99%

Overlapping computation and communication: Barrier algorithms and ConnectX-2 CORE-Direct capabilities

Graham

Poole

Shamis

et al. 2010

2010 IEEE International Symposium on Parallel &Amp; Distributed Processing, Workshops and PHD Forum (IPDPSW)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The newest generation of InfiniBand [24] adapters from Mellanox provide the Collective Offload Resource Engine, CORE-Direct [25]. This adds a management queue to the standard InfiniBand queue pair.…”

Section: Experimental Evaluationmentioning

confidence: 99%

Protocols for Fully Offloaded Collective Operations on Accelerated Network Adapters

Schneider

Hoefler

Grant

et al. 2013

2013 42nd International Conference on Parallel Processing

View full text Add to dashboard Cite

Abstract-With each successive generation, network adapters for high-performance networks are becoming more powerful and feature rich. High-performance NICs can now provide support for performing complex group communication operations on the NIC without any host CPU involvement. Several "offloading interfaces" have been designed with the collective communications goal being the complete offloading of arbitrary communication patterns.In this work, we analyze the offloading model offered in the Portals 4 specification in detail. We perform a theoretical analysis based on abstract communication graphs and show several protocols for implementing offloaded communication schedules. Based on our analysis, we propose and implement an extension to the portals 4 specification that enables offloading any communication pattern completely to the NIC. Our measurements with several advanced communication algorithms confirm that the enhancements provide good overlap and asynchronous progress in practical settings. Altogether, we demonstrate a complete and simple scheme for implementing arbitrary offloaded communication algorithms and hardware. Our protocols can act as a blueprint for the development of communication hardware as well as middleware while optimizing the whole communication stack. I. MOTIVATIONMoore's law is still going strong despite the end of frequency and Dennard scaling. CPU and chip vendors have managed to maintain Moore's law by going broad, i.e., duplicate functional units (e.g., cores or vector units) and/or add new functionality to chips. Thus, several microprocessor vendors began extending core functionalities of chips. For example, functionalities that were traditionally in a north bridge, such as a memory controller, are now commonly included in main CPUs. Similarly, networking chips have become more powerful and are to be integrated into next-generation CPUs.The growing number of cores per network endpoint increases the requirements for the network and memory interfaces. Modern multi-core CPUs already scale the number of memory controllers with the cores, similarly, network interfaces may follow. High-performance networks provide much more complex functionality than memory controllers. Thus, it seems reasonable to devote some silicon to performing advanced functions. The most important parameters of today's networks are latency, bandwidth, and message-rate. Therefore, modern networks are highly tuned to provide high performance for these metrics. However, many algorithms from scientific computing and other fields, such as databases, operating systems, and financial computations, use advanced communication algorithms over sets of processes. These are often called "collective communications" in high-performance computing (HPC) and they are important to many types of applications. Their increased complexity over standard point-to-point communications and participation of multiple processes make them important to overall communication performance, as well as a prime target for efforts to enhance network pe...

show abstract

“…Network Offload Architectures Some newer network architectures such as Portals IV [7] or CORE-Direct [14] allow to offload collective operations to the network device. This enables faster execution (messages do not need to travel to the CPU) and isolation (computations on the CPU and collective communications do not interfere and can progress independently).…”

Section: T Hoefler D Moormentioning

confidence: 99%

“…To offload a collective operation to a network device, one copies some state (e.g., a set of triggers [7] or a set of management queue entries [14]) that models the execution schedule to the device. The device then generates messages based on arriving messages from other processes and the local state without CPU involvement.…”

Section: T Hoefler D Moormentioning

confidence: 99%

Energy, Memory, and Runtime Tradeoffs for Implementing Collective Communication Operations

2014

JSFI

View full text Add to dashboard Cite

ConnectX-2 InfiniBand Management Queues: First Investigation of the New Support for Network Offloaded Collective Operations

Cited by 35 publications

References 9 publications

Overlapping computation and communication: Barrier algorithms and ConnectX-2 CORE-Direct capabilities

Overlapping computation and communication: Barrier algorithms and ConnectX-2 CORE-Direct capabilities

Protocols for Fully Offloaded Collective Operations on Accelerated Network Adapters

Energy, Memory, and Runtime Tradeoffs for Implementing Collective Communication Operations

Contact Info

Product

Resources

About