Machine-learning-aided cognitive reconfiguration for flexible-bandwidth HPC and data center networks [Invited]

Abstract

This paper proposes a machine-learning (ML)-aided cognitive approach for effective bandwidth reconfiguration in optically interconnected datacenter/high-performance computing (HPC) systems. The proposed approach relies on a Hyper-X-like architecture augmented with flexible-bandwidth photonic interconnections at large scales using a hierarchical intra/inter-POD photonic switching layout. We first formulate the problem of the connectivity graph and routing scheme optimization as a mixed-integer linear programming model. A two-phase heuristic algorithm and a joint optimization approach are devised to solve the problem with low time complexity. Then, we propose an ML-based end-to-end performance estimator design to assist the network control plane with intelligent decision making for bandwidth reconfiguration. Numerical simulations using traffic distribution profiles extracted from HPC applications traces as well as random traffic matrices verify the accuracy performance of the ML design estimator ( &lt;<!-- 9 % error) and demonstrate up to 5 × throughput gain from the proposed approach compared with the baseline Hyper-X network using fixed all-to-all intra/inter-portable data center interconnects.

Reconfigurable Optical Datacom Networks by Self-supervised Learning

“…The upper bound of Alizadeh Web Search distribution [2] was used to determine the sizes of flows. To emulate various traffic patterns, we used both synthetic traffic matrices and real HPC application traces, i.e., algebraic multi-grid (AMG), center for exascale simulation of advanced reactors (CESAR) and FFT [6,7]. The two synthetic traffic matrices were generated by randomly selecting 30% and 50% of the ToR pairs to generate demands following a uniform distribution.…”

“…However, such an approach cannot scale easily and has been demonstrated only in small-scale DC systems. Our previous work in [6] proposed a cognitive reconfiguration policy relying on performance (i.e., latency, packet loss rate) estimations by deep neural network (DNN) models. Nevertheless, training the DNN models requires collecting a large amount of performance data, introducing non-negligible operation overheads.…”