Cray XT4 and Seastar 3-D Torus Interconnect

Abts, Dennis

doi:10.1007/978-0-387-09766-4_22

Cited by 11 publications

(3 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Analysis of world experience in designing custom interconnection networks for high-end supercomputers, primarily the IBM BlueGene L/P/Q series [6,8,11] and Cray SeaStar/Gemini [1,5,7], simulation results [20] allowed to design the principles of operation of the first generation of the high-performance Angara interconnect (Angara G1) [21]. Series-produced Angara G1 hardware was presented in 2013.…”

Section: Introductionmentioning

confidence: 99%

The High Performance Interconnect Architecture for Supercomputers

2023

JSFI

View full text Add to dashboard Cite

In this paper, we introduce the design of an advanced high-performance interconnect architecture for supercomputers. In the first part of the paper, we consider the first generation high-performance Angara interconnect (Angara G1). The Angara interconnect is based on the router ASIC, which supports a 4D torus topology, a deterministic and an adaptive routing, and has the hardware support of the RDMA technology. The interface with a processor unit is PCI Express. The Angara G1 interconnect has an extremely low communication latency of 850 ns using the MPI library, as well as a link bandwidth of 75 Gbps. In the paper, we present the scalability performance results of the considered application problems on the supercomputers equipped with the Angara G1 interconnect. In the second part of the paper, using research results and experience we present the architecture of the advanced interconnect for supercomputers (G2). The G2 architecture supports 6D torus topology, the advanced deterministic and zone adaptive routing algorithms, and a low-level interconnect operations including acknowledgments and notifications. G2 includes support for exceptions, performance counters, and SR-IOV virtualization. A G2 hardware is planned in the form factor of a 32-port switch with the QSFP-DD connectors and a two-port low profile PCI Express adapter. The switches can be combined to 4D torus topology. We show the performance evaluation of an experimental FPGA prototype, which confirm the possibility of implementing the proposed advanced high performance interconnect architecture.

show abstract

Section: Introductionmentioning

confidence: 99%

The High Performance Interconnect Architecture for Supercomputers

2023

JSFI

View full text Add to dashboard Cite

show abstract

“…Typically, virtual channels [16] are introduced to break the cyclic dependencies for the ring of an arbitrary dimension in a torus [9,10,12,13,17]. While the way virtual channels are allocated may bring unbalanced virtual channel utilization.…”

Section: Introductionmentioning

confidence: 99%

“…Torus networks have lots of superior architecture proprieties, such as topology regularity, high path diversity, low node degree and linear scalability cost. Over the past 30 years, torus networks are widely used in commercial supercomputers and datacenter networks-examples include SGI Origin 2000 [1], Cray T3E [9], Cray XT series (3-D torus) [10], IBM Blue Gene L/Q (5-D torus) [11], Fujistu K computer (6-D mesh/torus) [12], Aton2 (3-D torus) [13] and Eurotech Aurora Datacenter (3-D torus) [14,15].…”

Section: Introductionmentioning

confidence: 99%

Balancing virtual channel utilization for deadlock-free routing in torus networks

Xiang

Wang

2015

J Supercomput

View full text Add to dashboard Cite

Torus networks have been widely used by modern commercial supercomputers due to low node degree and linear scalability cost. Meanwhile, the ring in each dimension of a torus creates cyclic channel dependencies, which pose challenges to the design of deadlock-free routing and virtual channel allocation schemes. Existing virtual channel allocation schemes such as dateline employ two virtual channels to avoid deadlock in a ring. While the unbalanced utilization of virtual channels caused by these deadlock avoidance schemes brings lots of performance pathologies. This paper focuses on improving deadlock-free routing algorithms in torus by balancing virtual channel utilization. First, we propose BVCU, a novel virtual channel allocation scheme that can balance the utilization of the two virtual channels used to prevent deadlock. The technique is deadlock free and can achieve almost perfect balanced utilization. Second, we propose a balanced dimension-order routing (BDOR) to provide deadlock-free deterministic routing for n-D torus by combining BVCU with dimension-order routing. Also, a balanced fully adaptive routing (BFAR) is proposed. These routing algorithms yield higher performance than existing solutions as they achieve a proper balanced utilization of virtual channels. Finally, simulation results show that the proposed methods achieve up to 16 % throughput improvement.

show abstract

Conditional forwarding: simple flow control to increase adaptivity for fully adaptive routing algorithms

Wang

Shen

2016

J Supercomput

View full text Add to dashboard Cite

Cray XT4 and Seastar 3-D Torus Interconnect

Cited by 11 publications

References 11 publications

The High Performance Interconnect Architecture for Supercomputers

The High Performance Interconnect Architecture for Supercomputers

Balancing virtual channel utilization for deadlock-free routing in torus networks

Conditional forwarding: simple flow control to increase adaptivity for fully adaptive routing algorithms

Contact Info

Product

Resources

About