Cost-Efficient Dragonfly Topology for Large-Scale Systems

Abstract

Evolving technology and increasing pin-bandwidth motivate the use of high-radix routers to reduce the diameter, latency, and cost of interconnection networks. This migration from low-radix to high-radix routers is demonstrated with the recent introduction of high-radix routers and they are expected to impact networks used in large-scale systems such as multicomputers and data centers. As a result, a scalable and a cost-efficient topology is needed to properly exploit high-radix routers.High-radix networks require longer cables than their low-radix counterparts. Because cables dominate network cost, the number of cables, and particularly the number of long, global cables should be minimized to realize an efficient network. In this paper, we introduce the dragonfly topology which uses a group of high-radix routers as a virtual router to increase the effective radix of the network. With this organization, each minimally routed packet traverses at most one global channel. By reducing global channels, a dragonfly reduces cost by 20% compared to a flattened butterfly and by 52% compared to a folded Clos network in configurations with ≥ 16K nodes.The paper also introduces two new variants of global adaptive routing that enable load-balanced routing in the dragonfly. Each router in a dragonfly must make an adaptive routing decision based on the state of a global channel connected to a different router. Because of the indirect nature of this routing decision, conventional adaptive routing algorithms give degraded performance. We introduce the use of selective virtual-channel discrimination and the use of credit round-trip latency to both sense and signal channel congestion. The combination of these two methods gives throughput and latency that approaches that of an ideal adaptive routing algorithm. SummaryThe interconnection network plays a critical role in the cost and performance of a scalable multiprocessor. Previous interconnection networks have been built with low-radix routers -i.e. routers with a small number of ports. As a result, these networks used low-radix topologies such as 2-D or 3-D mesh or torus networks. Examples of machines employing such networks include the Cray T3D, T3E, and XT3. Earlier work [4,2] showed that, for the packaging and technology constraints of the 80s and 90s, low-radix networks provide optimal latency for a given cost. This was true with the relatively low pin bandwidth available during the 80s and the early 90s. It is no longer the case.Over the past 20 years, the pin bandwidth of router chips has increased by approximately an order of magnitude every 5 years ( Figure 1) -a rate very similar to Moore's Law. This increase in bandwidth is a result of both an increase in the signaling rate and an increase in the number of signals. High-radix routers have been shown to take advantage of this increasing bandwidth by dividing the bandwidth into larger number of narrow ports [11]. In contrast, low-radix routers divide the bandwidth into a smaller number of wide ports. The Cray Black...

A case study in using massively parallel simulation for extreme-scale torus network codesign

“…This property can be leveraged to balance load across network channels. Unlike other hierarchical networks like the butterfly and dragonfly [7,8], torus networks have a higher average number of hop counts traversed.…”

“…For serial simulation, Dally et al developed Booksim which is a cycle-accurate network simulation framework, to study the performance of the torus network and others topologies in comparison to a dragonfly network topology [7,35]. While Booksim provides the support for a number of network topologies, the maximum size of the network is limited to only 1024 nodes.…”

“…One viable option is to continue using k-ary, n-dimensional torus networks that have been extensively used in modern supercomputers such as the IBM Blue Gene (BG) series [3,4], Cray XT, and Cray XE series networks [5]. Another option is to use low-latency, low-diameter interconnect topologies (such as the dragonfly used by Cray XC30 system [6,7], flattened butterfly [8], or PERCS [9]) that will enable fast communication at large node counts while keeping power usage and physical construction parameters in check.…”

Generalized Hierarchical All-to-All Exchange Patterns

“…This concept includes, among others, all variations of tree interconnects (trees, fat trees [3], extended generalized fat trees (XGFTs) [4]) as well as hierarchical fullymeshed designs such as the dragonfly [1], [5]. All hierarchical topologies share a few key features.…”

HiNetSim: A Parallel Simulator for Large-Scale Hierarchical Direct Networks

“…To balance the scalability and performance, the hierarchical direct network architecture corresponding to the communication locality is widely used. In the hierarchical network, different hierarchies of the network often possesses different switching capacities, such as IBM BlueGene/Q 5D torus [2, 3] K tofu nested 6D torus [4] and Cray dragonfly [5]. Facing with different switching capacities, the network topology varies between different network hierarchies.…”