Mercury: A Simple Transport Layer Scheduler to Accelerate Distributed DNN Training

US-Byte: An Efficient Communication Framework for Scheduling Unequal-Sized Tensor Blocks in Distributed Deep Learning

“…Communication acceleration: Existing communication acceleration techniques include, but are not limited to: (1) leveraging high throughput and low latency communication links, such as RDMA [34], [35], [36], InfiniBand, Intel Omni-Path, and NVIDIA's NVLink 3 ; (2) utilizing message passing interface (MPI) and MPI-like implementations like OpenMPI 4 and Gloo [37]; (3) using high-performance communication collectives, such as NCCL 5 and BLink [38], which support efficient communication between GPUs and many popular deep learning frameworks; (4) reducing data communication during synchronization process, such as gradient quantization, compression and sparsification [39], [40], [41], [42], [43], [44]; (5) using stale parameter updates to reduce the number of synchronization parameters, such as parameter freezing [45], [46], [47], Round-Robin Synchronous Parallel [48] and Bounded Staleness Parallel [49]; (6) tuning deep learning hyper-parameters, such as AutoByte [50]; (7) minimize user-level overhead by conducting parameter aggregation at the transport layer [13]; (8) improving network-layer performance, such as networklevel flow scheduling [51], [52] and congestion control [53]. Communication scheduling: Due to the layer-wise and tensor-wise structure of DNNs, some works continuously explore to maximize the overlap of communication and computation.…”

“…There are various factors that limit the scaling efficiency of data parallelism, such as energy consumption [7], [8], dataset privacy [9], [10], [11], and imbalance of computing resources [12]. In addition, with large-scale datasets, models, and clusters, intensive data communication due to gradient aggregation among nodes introduces a huge additional time overhead for data parallelism and limits its scalability [4], [5], [13]. Consequently, communication performance has become a significant bottleneck in distributed DNN training.…”

“…Meanwhile, the execution order of these operations is determined by the underlying DAG. Since these two types of operations do not occupy the same resources, researchers find that communication can be overlapped with computation tasks in data parallelism to reduce the time of one iteration without affecting the convergence performance of the DNN [13]. Wait-free backward propagation (WFBP) has been adopted in most deep learning frameworks [17].…”