FA 18.4: a phase-tolerant 3.8 GB/s data-communication router for a multiprocessor supercomputer backplane

Abstract

Recent parallel processor supercomputer designs use an active backplane of routers to form the interconnections between processing elements [ 1, 21. Today, high-bandwidth interconnect systems capable of scaling to configurations with >500 processing nodes tend to use self-timed designs. This avoids clock distribution problems seen in large phase-sensitive synchronous systems. The BiCMOS routing component described here employs 200MHz clocked communication for large scalable parallel-processor supercomputer systems. This scheme eliminates need for clock edges phase-aligned across the clock distribution network. Additionally, router inputs accept data at any phase relationship to the receiving router internal clock.The router communicates with its nearest neighbor routers in both theX and Y directions, as well as with a processing node.Router-to-router connections are clocked. The router-to-processing-node connection is self-timed and operates a t any frequency up to the 200MHz router backplane frequency.Each proressing node contains a network interface component (NIC), a microprocessor, and memory. The processing node connection to the backplane is shown in Figure 1. To send a message from one processing node to another, the source processor includes a routing header (an X andY displacement) in the first two 16b words of the message. The routing header specifies the path through the backplane to the final destination. The message is passed to arouter that is part ofthe active backplane. Routers pass messages to nearest neighbors first along the X-direction (until the X-displacement has been reached), and then along the Y-direction (until the Y-displacement has been reached) similarly to other 2D deterministic routers [31. The destination router passes the message (minus the header) to its processing node.Each router communicates with its neighbors via unidirectional point-to-point port connections for inputs and outputs ( Figure 2). The ports are grouped in inbouncUoutbound pairs. Each unidirectional port consists ofdata, parity, control, transmit clock, and reference voltage. Input buffers are differential amplifiers with one input connected to the data pin and the other connected to the reference voltage generated by the sending router, reducing common-mode noise a t the input.The receiving router latches incoming data using both edges of the data-transmit clock. The transmit clock transitions a t the same time as the data. The receiver uses a delay-locked loop to center received transmit clock on received data. Centering of data clocks maximizes jitter margins for data clock edges with resptbct t o incoming data. Figure 3 shows a simplified diagram of the delay-locked loop. The input data transmit clock goes through two equal delay circuits. The centered output is taken from the middle of the two delay circuits. The output of the second delay block is fed back and compared to the input signal. Ifthe combined delay is less than the input clock high time then the control to thedelay circuit increases each blockde...

The IA-64 Itanium processor cartridge

“…The data bus can operate in a doublepumped (source synchronous 3,4 ) transfer mode. That is, data transfers occur twice for every bus clock cycle (see the example in Figure 5).…”

The performance impact of flexibility in the Stanford FLASH multiprocessor

“…Any time a message enters the network, it is charged a fixed network transit latency. This latency is based on the average transit time for a two-dimensional mesh network having a per-hop fallthrough time of 40 ns [Intel94]. For our 16-processor simulations, the average message requires latency equivalent to one hop to both enter and exit the network, 2.6 hops of network transit, and 3 cycles of network header information, yielding an average transit time of 220 ns, or 22 cycles.…”

A 8-Gb/s/pin current mode multi-level simultaneous bidirectional I/O

“…In chip-to-chip communication, increasing the bandwidth per wire enhances system performance due to limited number of pins [1] [2]. Simultaneous Bidirectional (SBD) signaling was previously introduced to allow simultaneous data transmission in two directions over one wire, doubling the effective bandwidth per wire over a point-to-point unidirectional transmission [3].…”