Thrifty interconnection network for HPC systems

Abstract

We propose Thrifty Interconnection Network (TIN), where the network links are activated and de-activated dynamically to save power with little or no overhead by using inherent system events to overlap the link activation or de-activation time. Our simulation results on a set of real world HPC workload traces show on average 35% network power reduction.Power is a critical problem in modern supercomputing systems [1]. A large-scale supercomputer, running High Performance Computing (HPC) workloads, can include hundreds of thousands of processing nodes connected via a large packet-switched interconnection network. A closer look into these systems reveals that the power consumption of interconnection links (including link controllers) constitutes not only a majority of the power of the switches, but also a substantial percentage of the total system power. For instance, the links in an IBM 8-port 12X switch can take 64% of the switch power. The power consumption of the interconnection network in HPC systems can contribute to around 30% the total system power [2]. Current high-speed links in the interconnection networks require continuous pulse transmission to keep both ends synchronized, even when no data is transmitting. Therefore, the average power consumption of such links is almost identical to their worst-case power consumption. Furthermore, network subsystem designers often over-provision the network capacity, with higher power consumption, to meet performance commitments and to avoid network congestion.We observe that HPC workloads rarely operate all system elements at their maximum capacity simultaneously. For example, infrequent communication patterns exhibited by may HPC workloads allows provisioned network links to stay idle for most of the time. Fig. 1 illustrates a data communication path in an HPC system from a sender compute node (compute node 0) to a receiver compute node (compute node M).Our approach to saving network power is based on an observation that there can be a significant delay from when a switch sees the first command of a data transfer to when it sends out the first data packet. We propose the Thrifty Interconnection Network (TIN). TIN includes a hardware approach that overlaps inherent system events with link transition delay for significant network power reduction without noticeable network performance overhead. Fig. 2 illustrates a high-level flowchart of providing link services in the thrifty network. Fig. 3 illustrates the hardware support for the thrifty network link management. Note that our proposal differs from prior art in that we do not use prediction. It is also independent on the network topology.TIN also includes a software extension that uses two softwareinitiated commands as hints to activate and release links. With this two commands, MPI programs or run time systems can be instrumented to identify longer MPI communication phase to reduce the number of link power state transitions.We use an in-house simulator called MARS (MPI Application Relay network Simulator) to simulate...

Energy proportional datacenter networks

“…Other approaches [18] have focused on the MPI run-time system to activate or deactivate links involved in the traffic pattern. This approach requires detailed knowledge of the traffic pattern a priori.…”

Análise da Pegada de Carbono em Aplicações Paralelas

“…Contudo, essas inovac ¸ões vêm com um aumento significativo no consumo de energia, o que, por sua vez, eleva a pegada de carbono e desafia a sustentabilidade ambiental desses sistemas. Essa pegada se manifesta em duas fases distintas: inicialmente, durante a fabricac ¸ão do hardware, onde é conhecida como carbono incorporado, e posteriormente, na fase operacional do sistema, referida como carbono operacional [Li et al 2023].…”

Exploiting variability for energy optimization of parallel programs

“…Most existing techniques [17,20,21,30] exploit the idea that the presence of slack, occurring when tasks wait at synchronization points, indicates that DVFS can be applied on those cores waiting for external events. Our survey of these state-of-the-art techniques indicates that: 1) results are demonstrated on codes that use static domain decomposition and exhibit static load imbalance; 2) decisions are made based on predictions of program running time; and 3) DVFS is always controlled per core.…”