Congestion avoidance on manycore high performance computing systems

Abstract

Efficient communication is a requirement for application scalability on High Performance Computing systems. In this paper we argue for incorporating proactive congestion avoidance mechanisms into the design of communication layers on manycore systems. This is in contrast with the status quo which employs a reactive approach, e.g. congestion control mechanisms are activated only when resources have been exhausted. We present a core stateless optimization approach based on open loop end-point throttling, implemented for two UPC runtimes (Cray and Berkeley UPC) and validated on InfiniBand and the Cray Gemini networks. Microbenchmark results indicate that throttling the number of messages in flight per core can provide up to 4X performance improvements, while throttling the number of active cores per node can provide additional 40% and 6X performance improvement for UPC and MPI respectively. We evaluate inline (each task makes independent decisions) and proxy (server) congestion avoidance designs. Our runtime provides both performance and performance portability. We improve allto-all collective performance by up to 4X and provide better performance than vendor provided MPI and UPC implementations. We also demonstrate performance improvements of up to 60% in application settings. Overall, our results indicate that modern systems accommodate only a surprisingly small number of messages in flight per node. As Exascale projections indicate that future systems are likely to contain hundreds to thousands of cores per node, we believe that their networks will be underprovisioned. In this situation, proactive congestion avoidance might become mandatory for performance improvement and portability.

Exploiting communication concurrency on high performance computing systems

“…For large messages, parallelization does not improve performance when compared to the original setting, yet there is a noticeable difference from peak bandwidth. In this case orthogonal concurrency throttling techniques as described by Luo [17] are required. Note that due to decoupling the communication into a standalone subsystem, these techniques are easy to implement in our architecture.…”

“…For well balanced applications, there usually exists communication concurrency, even enough to cause congestion [17]. In this case throttling the spatial concurrency of communication improves performance.…”

Maximizing Communication Overlap with Dynamic Program Analysis

“…With strong scaling, messages become large at low concurrency. As indicated by Luo et al [27] on the networks we use (Aries and In-finiBand) issuing simultaneously a large number of large messages degrades performance.…”

Network Endpoints for Clusters of SMPs

“…While there is little related work on the endpoints as introduced in [10], a large body of work exists on the exploitation of shared memory nodes within MPI or other parallel programming languages like UPC [7], [9], [13]. Many of these papers focus on optimizing various communication primitives by means of a shared memory region.…”

“…The main program (Line 3) creates the PAMI client and a number of contexts. Our simple example creates a thread to handle each context (lines [11][12][13]. Each thread receives the context ID as an argument (not shown for brevity).…”