Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning

Abstract

In distributed training of deep neural networks, parallel minibatch SGD is widely used to speed up the training process by using multiple workers. It uses multiple workers to sample local stochastic gradient in parallel, aggregates all gradients in a single server to obtain the average, and update each worker's local model using a SGD update with the averaged gradient. Ideally, parallel mini-batch SGD can achieve a linear speed-up of the training time (with respect to the number of workers) compared with SGD over a single worker. However, such linear scalability in practice is significantly limited by the growing demand for gradient communication as more workers are involved. Model averaging, which periodically averages individual models trained over parallel workers, is another common practice used for distributed training of deep neural networks since (Zinkevich et al. 2010) (McDonald, Hall, and Mann 2010). Compared with parallel mini-batch SGD, the communication overhead of model averaging is significantly reduced. Impressively, tremendous experimental works have verified that model averaging can still achieve a good speed-up of the training time as long as the averaging interval is carefully controlled. However, it remains a mystery in theory why such a simple heuristic works so well. This paper provides a thorough and rigorous theoretical study on why model averaging can work as well as parallel mini-batch SGD with significantly less communication overhead.1 Equivalently, we can let the server update its solution using the averaged gradient and broadcast this solution to all local workers. Another equivalent implementation is to let each worker take a single SGD step using its own gradient and send the updated local solution to the server; let the server calculate the average of all workers' updated solutions and refresh each worker's local solution with the averaged version. arXiv:1807.06629v3 [math.OC]

Qsparse-Local-SGD: Distributed SGD With Quantization, Sparsification, and Local Computations

“…We emphasize that unlike [YYZ18,Sti19], which only consider local computation, we combine quantization and sparsification with local computation, which poses several technical challenges; e.g., see proofs of Lemma 4, 5, 6.…”

“…[WHHZ18] analyzed error compensation for QSGD, without Top k sparsification while focusing on quadratic functions. Another approach for mitigating the communication bottlenecks is by having infrequent communication, which has been popularly referred to in the literature as iterative parameter mixing, see [Cop15], and model averaging, see [Sti19,YYZ18,ZSMR16] and references therein. Our work is most closely related to and builds upon the recent theoretical results in [AHJ + 18, SCJ18,Sti19,YYZ18].…”

“…Another approach for mitigating the communication bottlenecks is by having infrequent communication, which has been popularly referred to in the literature as iterative parameter mixing, see [Cop15], and model averaging, see [Sti19,YYZ18,ZSMR16] and references therein. Our work is most closely related to and builds upon the recent theoretical results in [AHJ + 18, SCJ18,Sti19,YYZ18]. [SCJ18] considered the analysis for the centralized Top k (among other sparsifiers), and [AHJ + 18] analyzed a distributed version with the assumption of closeness of the aggregated Top k gradients to the centralized Top k case, see Assumption 1 in [AHJ + 18].…”

Adaptive Federated Learning in Resource Constrained Edge Computing Systems

“…Multiple local updates before aggregation is possible in the bound derived in [26], but the number of local updates varies based on the thresholding procedure and cannot be specified as a given constant. Concurrently with our work, bounds with a fixed number of local updates between global aggregation steps are derived in [32], [33]. However, the bound in [32] only works with i.i.d.…”

Orchestrating the Development Lifecycle of Machine Learning-based IoT Applications

“…By varying the batch size [124,312,373], this method is effective in reducing the communication cost without too much accuracy loss. In the next paragraph, we will discuss more about the parallel SGD algorithms [226,316,375,383,399] for improving the communication efficiency, which can be seen as one way of improving the performance of data parallelism. Another type of data parallel that addresses the memory limit on single GPU is spatial parallelism [167].…”