On the Utility of Gradient Compression in Distributed Training Systems

Abstract

Rapid growth in data sets and the scale of neural network architectures have rendered distributed training a necessity. A rich body of prior work has highlighted the existence of communication bottlenecks in synchronous data-parallel training. To alleviate these bottlenecks, the machine learning community has largely focused on developing gradient and model compression methods. In parallel, the systems community has adopted several High Performance Computing (HPC) techniques to speed up distributed training. In this work, we evaluate the efficacy of gradient compression methods and compare their scalability with optimized implementations of synchronous data-parallel SGD. Surprisingly, we observe that due to computation overheads introduced by gradient compression, the net speedup over vanilla data-parallel training is marginal, if not negative. We conduct an extensive investigation to identify the root causes of this phenomenon, and offer a performance model that can be used to identify the benefits of gradient compression for a variety of system setups. Based on our analysis, we propose a list of desirable properties that gradient compression methods should satisfy, in order for them to provide a meaningful end-to-end speedup.

A Field Guide to Federated Optimization

“…For example, many distributed systems use all-reduce style techniques to aggregate gradients or local model updates across compute nodes [198]. In such systems, compression techniques that are not compatible with all-reduce may provide less communicationefficiency, despite their higher compression ratio [5,253,258]. In such settings, it is important to make sure that the compression operation commutes with addition.…”

“…In particular, if we force all clients to use τ -fold larger mini-batch size in synchronous SGD algorithm, then this algorithm can also save τ times communication, which is the same as local-update algorithms. We refer to this algorithm as large-batch synhcronous SGD 5 . It has been well understood (see [66]) that the worst-case error of large-batch synchronous SGD is:…”

Distributed Learning With Sparsified Gradient Differences

“…An efficient solution should significantly reduce the total number of bits used in a given transmission without impacting rapid convergence to a good estimate of θ. In order to provide an end-to-end speedup, a list of properties that compression methods should satisfy are proposed in [27]. Quantization is a popular data compression approach which aims to approximate some quantity using a smaller number of bits to simplify processing, storage and analysis.…”