Optimized InfiniBand^TM fat‐tree routing for shift all‐to‐all communication patterns

Abstract

SUMMARYClustered systems have become a dominant architecture of scalable high-performance super computers. In these large-scale computers, the network performance and scalability is as critical as the computenodes speed. InfiniBand TM has become a commodity networking solution supporting the stringent latency, bandwidth and scalability requirements of these clusters. The network performance is also affected by its topology, packet routing and the communication patterns the distributed application exercises. Fattrees are the topology structures used for constructing most large clusters as they are scalable, maintain cross-bisectional-bandwidth (CBB), and are practical to build using fixed-arity switches. In this paper, we propose a fat-tree routing algorithm that provides a congestion-free, all-to-all shift pattern leveraging on the InfiniBand TM static routing capability. The algorithm supports partially populated fat-trees built with switches of arbitrary number of ports and CBB ratios. To evaluate the proposed algorithm, detailed switch and host simulation models were developed and multiple fabric topologies were run. The results of these simulations as well as measurements on real clusters show an improvement in all-to-all delay by avoiding congestion on the fabric.

A Novel Query Caching Scheme for Dynamic InfiniBand Subnets

“…The Fat-Tree routing algorithm (FTree) is a topology-aware routing algorithm for Fat-Tree topologies [104,105,108]. We have modified the Fat-Tree routing algorithm for the needs of Paper III, Paper IV and Paper V. In this section we are going to explain how the basic routing algorithm has been implemented in OpenSM 9 ; FTree first discovers the network topology and each switch is marked with a tuple that identifies its location in the topology.…”

Congestion avoidance on manycore high performance computing systems

“…Dally [12] pioneered the concept of wormhole routing and his work has been since extended with congestion free routing alternatives on a very large variety of network topologies. For example, Zahavi et al [27] recently proposed a fat-tree routing algorithm that provides a congestion-free all-to-all shift pattern for the InfiniBand static routing.…”

Improving Performance of Deterministic Single-Path Routing on 2-Level Generalized Fat-Trees

“…Existing deterministic single-path routing algorithms for ( + , ) are either the Source-mod-k routing (S-m-k) [6], [8], [9] or the Destination-mod-k routing (D-m-k) [3], [9], [12]. A good summary of routing in generalized fat-trees can be found in [9].…”

“…We will use the notion ( + , ) to denote such a 2-level generalized fat-tree. The cross-bisection bandwidth (CBB) ratio [12] of ( + , ) is : when = , ( + , ) is a full bisection bandwidth fat-tree; when < , ( + , ) is a slimmed fat-tree; when > , ( + , ) is a fatted fat-tree. Examples of full bisection bandwidth, slimmed, and fatted fat-trees are shown in Figures 1 (b), (c), and (d), respectively.…”

“…In this routing scheme, one path is used to carry all traffics from a source to a destination. Existing single-path deterministic routing for generalized fat trees include Sourcemod-k routing [6], [8], [9] and Destination-mod-k routing [3], [9], [12]. As will be shown later, all of these existing routing schemes are not ideal for many common communication patterns.…”