Protocols for Fully Offloaded Collective Operations on Accelerated Network Adapters

Abstract

Abstract-With each successive generation, network adapters for high-performance networks are becoming more powerful and feature rich. High-performance NICs can now provide support for performing complex group communication operations on the NIC without any host CPU involvement. Several "offloading interfaces" have been designed with the collective communications goal being the complete offloading of arbitrary communication patterns.In this work, we analyze the offloading model offered in the Portals 4 specification in detail. We perform a theoretical analysis based on abstract communication graphs and show several protocols for implementing offloaded communication schedules. Based on our analysis, we propose and implement an extension to the portals 4 specification that enables offloading any communication pattern completely to the NIC. Our measurements with several advanced communication algorithms confirm that the enhancements provide good overlap and asynchronous progress in practical settings. Altogether, we demonstrate a complete and simple scheme for implementing arbitrary offloaded communication algorithms and hardware. Our protocols can act as a blueprint for the development of communication hardware as well as middleware while optimizing the whole communication stack. I. MOTIVATIONMoore's law is still going strong despite the end of frequency and Dennard scaling. CPU and chip vendors have managed to maintain Moore's law by going broad, i.e., duplicate functional units (e.g., cores or vector units) and/or add new functionality to chips. Thus, several microprocessor vendors began extending core functionalities of chips. For example, functionalities that were traditionally in a north bridge, such as a memory controller, are now commonly included in main CPUs. Similarly, networking chips have become more powerful and are to be integrated into next-generation CPUs.The growing number of cores per network endpoint increases the requirements for the network and memory interfaces. Modern multi-core CPUs already scale the number of memory controllers with the cores, similarly, network interfaces may follow. High-performance networks provide much more complex functionality than memory controllers. Thus, it seems reasonable to devote some silicon to performing advanced functions. The most important parameters of today's networks are latency, bandwidth, and message-rate. Therefore, modern networks are highly tuned to provide high performance for these metrics. However, many algorithms from scientific computing and other fields, such as databases, operating systems, and financial computations, use advanced communication algorithms over sets of processes. These are often called "collective communications" in high-performance computing (HPC) and they are important to many types of applications. Their increased complexity over standard point-to-point communications and participation of multiple processes make them important to overall communication performance, as well as a prime target for efforts to enhance network pe...

Exploiting Offload Enabled Network Interfaces

“…III. Schneider et al [22] discuss about protocols for fully offloaded collectives, however, their protocol requires synchronization among the involved nodes. Barrett et al [8] propose an offloaded version of the rendezvous protocol based on Portals 4 triggered operations, requiring CPU intervention in the unexpected message case.…”

“…In the sender-initiated version, a control message is sent to the receiver that will reply when the matching receive will be posted (and thus the receiver buffer will be ready). In the receiver-initiated version [22], the receiver has to signal to the sender when it is able to receive the message. Without loss of generality, in this work we consider only the sender-initiated variant of this protocol, since the receiver-initiated one can be implemented similarly.…”

A Survey: Runtime Software Systems for High Performance Computing

“…A number of convenience constructs, such as parallel threaded loops and reduction operations are also provided. The remote operation is built on top of Portals4 library [14]. Qthreads execute on POSIX-compliant machines and have been tested on Linux, Solaris, and Mac OS using GNU, Intel, PGI, and Tilera compilers.…”

Towards A Data Centric System Architecture: SHARP

“…Several APIs have been proposed for offloading collective operation management to the HCA. This includes the Mellanox's CORE-Direct [13], protocol, Portal 4.0 triggered operations [7], and an extension to Portals 4.0 [29]. All these support protocols that use end-point management of the collective operations, whereas in the current approach the end-points are involved only in collective initiation and completion, with the switching infrastructure supporting the collective operation management.…”