The BlackWidow High-Radix Clos Network

Abstract

This paper describes the radix-64 folded-Clos network of the Cray BlackWidow scalable vector multiprocessor. We describe the BlackWidow network which scales to 32K processors with a worst-case diameter of seven hops, and the underlying high-radix router microarchitecture and its implementation. By using a high-radix router with many narrow channels we are able to take advantage of the higher pin density and faster signaling rates available in modern ASIC technology. The BlackWidow router is an 800 MHz ASIC with 64 18.75Gb/s bidirectional ports for an aggregate off-chip bandwidth of 2.4Tb/s. Each port consists of three 6.25Gb/s differential signals in each direction. The router supports deter-ministic and adaptive packet routing with separate buffering for request and reply virtual channels. The router is organized hierarchically [13] as an 8×8 array of tiles which simplifies arbitration by avoiding long wires in the arbiters. Each tile of the array contains a router port, its associated buffering, and an 8×8 router subswitch. The router ASIC is implemented in a 90nm CMOS standard cell ASIC technology and went from concept to tapeout in 17 months.

Express Cube Topologies for on-Chip Interconnects

“…The YARC router [17] employed an 8x8 grid of switches to implement a radix-64 router. Each switch was connected to other switches in a given row via a dedicated bus, making it a one-to-many configuration similar to MECS.…”

Occupancy Sampling for Terabit CEE Switches

“…Ideally, QCN at an input buffer should detect overload, mark and throttle the culprit flows and, by using an appropriate Q eq , keep the input buffer backlog below the PFC-high threshold. 3 As we show next, naively applying standard QCN at inputs may severely reduce or even nullify the expected performance improvement versus a PFC-only solution.…”

“…Crossbar chips with several tens of 10G and 40G Ethernet ports are now emerging in the market, while 100G Ethernet technology is also underway [3], [4]. Such highradix crossbars enable larger switches and switching fabrics using fewer crossbar ASICs.…”

“…3 for a PFC-only baseline test with QCN disabled, each flow receives a bandwidth share of 10 4 = 2.5 Gb/s during the congestive phase. This is fair for 3 CNMs generated by CPs at inputs do not traverse the fabric; hence they neither consume fabric-internal bandwidth nor incur any additional delays. the K − 1 = 4 flows that share the 10G capacity of an output link.…”

Implementation and Evaluation of Network Interface and Message Passing Services for TianHe-1A Supercomputer

“…In order to reduce the hops in large-scale interconnect network, lower communication latency, and improve the reliability and flexibility of interconnect fabric, we used highradix routing techniques [3] to design NRC. As shown in Figure 1, each NRC contains 16 symmetric interconnect switch ports.…”