What package improvements are required for dense, high aggregate bandwidth buses running at data rates beyond 10 Gb/s per pin, and when might optical interconnects on the board be required? We present a study of distance and speed limits for electrical on-board module-to-module links with an eye to answering these questions. Detailed electrical link models have been validated with active, high-speed differential bus measurements utilizing a 16-channel link chip with programmable equalization and a per-channel data rate of up to 11 Gb/s. Electrical signalling limits were then determined by extrapolating our models to higher speeds, and these limits were compared to the results of work on on-board optical interconnects.
IntroductionMulti-core, multi-socket servers and high-end switch systems are driving dense, multi-gigabit signaling. Early work on the OIF CEI-25 standard, aimed at specifying a parallel 20 to 25 Gb/s electrical interface for future 40 or 100 Gb/s optical modules, has shown that legacy channels are inadequate at speeds beyond ~17 to 20 Gb/s. At the same time, future high port-count switches and high-end servers will require hundreds to thousands of electrical links running at speeds ≥ 10 Gb/s to meet rising bandwidth demands.Although another study [1] has focused on two modules connected by flex, our study explores module-on-board packaging topologies seen in switches and servers where more than two modules are connected via high-bandwidth buses through a printed circuit board (PCB). We describe the link configurations and packaging technologies aimed at this application space, then show how each element in the electrical link was modeled, followed by model validation against passive hardware measurements. We then present active link measurements at 11 Gb/s and show the correlation with end-to-end link simulations. Finally, we use these hardware-correlated models in simulations to predict the performance of dense buses running at 25 Gb/s rates, and compare this to recent work [2] on on-board optical interconnects.