Optical Interconnects in Future Servers

Abstract

Optical interconnects are common in today's petascale supercomputers, and will become pervasive at the exascale during this decade. Technologies that can meet the challenging technological and economic requirements for the exascale will be reviewed. Optics in Petascale SupercomputersIn 2008, Roadrunner[1], the first petaflop supercomputer (i.e., >10 15 floating point operations per second) was constructed. The large number of microprocessor cores in such petascale machines must be interconnected with a high capacity communications network to permit efficient computation. Higher interconnect bandwidth will generally result in more efficient use of the microprocessors in real calculations. Prior to 2005, this network used electrical interconnects in essentially all supercomputers. Communication bitrates increased beyond ~2Gb/s starting in about 2005. For distances greater than about 20m, electrical interconnects were impractical and optics began to be used for these longer rack-to-rack interconnects (e.g., ASCI Purple[2]). For cost reasons, however, the shorter rack-to-rack interconnect in this machine was still electrical. By 2008, communication bitrates had increased to 5Gb/s (InfiniBand DDR), and the practical reach of electrical interconnects was shorter. At the same time, the cost of optics had decreased. As a result, for Roadrunner, all node-to-node communication was optical. However, in these examples, all the intra-node communication was electrical. Also, the conversion to optics was at the edge of the node/rack, requiring an electrical interconnect from the chip through several levels of packaging before conversion to photons. For machines with peak performance beyond 10PF, optics will play a far more important role. For example, in the planned POWER7-IH systems [3], all the board-to-board interconnects, will be optical. A major change in the packaging tightly integrates the optics onto the chip-level package. Optical transmitter and receiver modules are placed on the same ceramic Hub (switch) Module [4,5] , with each channel using a 10Gb/s signaling rate and 8b10b coding. Projections for future supercomputersBecause microprocessor clock speeds are not rapidly increasing, supercomputing performance improvements come from larger numbers of processing cores operating in parallel. For example [7], the #1 machine on the top500 list from June, 2002 was the NEC Earth-Simulator with 5120 single core processor chips, as compared to Roadrunner which has some 97,920 cores in 12,240 Cell Broadband Engine® processor chips plus 6,562 dual-core AMD Opteron® chips [8]. This increasing core and chip count drives increasing communication bandwidth at higher bitrates.Trends for optical interconnects in future supercomputers can be extrapolated from past growth. Historically, computational power has grown approximately 10x every 4 years [7], and this growth is expected to continue. Affordable supercomputing requires that the cost and power consumption of supercomputers grow more slowly than computational power. ...

How will photonic integrated circuits develop?

“…Much research and development has been, and is currently, focused on applying photonic integrated circuits (PICs) to chip-scale digital optical interconnects (OI) for high-density, high-throughput, computer communications [1][2][3][4][5][6][7][8] and networking [9]. In OI there is a specific goal of overcoming the performance limits of high-density metal interconnects and there has been an emphasis on developing monolithically integrated Silicon Photonic platforms to exploit the fabrication precision and device densities associated with Silicon ICs, as well as the potential for tight integration with Silicon electronic circuits.…”

Augmenting data rate performance for higher order modulation in triangular index profile multicore fiber interconnect

“…Space division multiplexing (SDM) has recently attracted attention as a potential means to overcome the imminent capacity crunch of short reach optical 2 transmission system [3]. The key issues for design of the forthcoming heterogeneous and bandwidth intensive OI are high fiber count and high density cable with minimum escalation in link cost and power budget [4,5]. The SDM technology based on multicore fibers (MCFs) has potential to thrust the data traffic capacity up to an unprecedented level [6].…”

A polymer waveguide-based 40 Gb/s optical bus backplane for board-level optical interconnects

“…Optical technologies can overcome the inherent disadvantages of copper-based interconnects when operating at high (> 5 Gb/s) data rates and offer increased bandwidths, reduced power consumption, immunity to electromagnetic interference and relaxed thermal management requirements [3,4]. Fibre-based optical interconnects are currently widely deployed for rack-to-rack communications and are now gradually penetrating into intra-rack communications [5,6]. Polymer multimode waveguides have attracted particular attention for use in board-level interconnections as they can be directly integrated onto standard printed circuit boards (PCBs) and offer relaxed alignment tolerances in the assembly and packaging of these hybrid opto-electronic (OE) boards [7][8][9].…”