TopoOpt: Optimizing the Network Topology for Distributed DNN Training

Abstract

We explore a novel approach for building DNN training clusters using commodity optical devices. Our proposal, called TOPOOPT, co-optimizes the distributed training process across three dimensions: computation, communication, and network topology. TOPOOPT uses a novel alternating optimization technique and a group theory-inspired algorithm to find the best network topology and routing plan, together with parallelization strategy, for distributed DNN training. To motivate our proposal, we measure the communication patterns of distributed DNN workloads at a large online service provider. Experiments with a 12-node prototype demonstrate the feasibility of TOPOOPT. Simulations on real distributed training models show that, compared to similar-cost Fat-tree interconnects, TOPOOPT reduces DNN training time by up to 3×.

Efficient neural network accelerators with optical computing and communication

“…As illustration of Fig. Motivated by the optical switch based topologies in [33], authors in [70] proposed a co-optimizes network topology and a parallelization strategy for ANN training system named TOPOOPT. The proposed scheme searches over the parallelization strategy space with a fixed topology, and returns the communication demands to the system.…”

Partitioning Distributed Compute Jobs with Reinforcement Learning and Graph Neural Networks

“…Recent years have seen a surge of interest in developing methods to distribute machine learning (ML) tasks across multiple devices (Ben-Nun and Hoefler, 2019;Mayer and Jacobsen, 2020). One approach has been to optimise the physical plane of the distributed cluster such as its compute and network devices and architectures (Parsonson et al, 2020;Khani et al, 2021;Wang et al, 2022;Ottino et al, 2022). In this work, we instead focus on optimising the virtual plane, which determines how physical layer resources are allocated to execute a job.…”

“…Figure 1: How the network overhead of six distributed deep learning jobs (encompassing object tracking, recommendation, natural language processing, and image recognition) increases with the number of workers used in Meta's GPU cluster (Wang et al, 2022).…”

“…of workers used to execute a job is increased, the per-worker computation demands decrease, but the overall communication overhead between workers grows (see Figure 1). This shifts the performance bottleneck away from the workers themselves and into the network connecting them, and brings additional challenges with managing varying traffic characteristics for different job types and parallelisation strategies (Wang et al, 2022;Parsonson et al, 2022a;Benjamin et al, 2021Benjamin et al, , 2022.…”

“…To address the communication bottleneck in distributed computing, recent works have sought to develop optical clusters (Benjamin et al, 2020;Ballani et al, 2020;Khani et al, 2021;Wang et al, 2022;Ottino et al, 2022); machines interconnected by optical switches (Parsonson et al, 2020;Gerard et al, 2020Gerard et al, , 2021. Compared to their electronic counterparts, optically switched networks offer orders of magnitude improvements in scalability, bandwidth, latency, and power consumption (Ballani et al, 2020;Zervas et al, 2018;Mishra et al, 2021) (see Section 3).…”