DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training

Abstract

The scale of deep learning nowadays calls for efficient distributed training algorithms. Decentralized momentum SGD (DmSGD), in which each node averages only with its neighbors, is more communication efficient than vanilla Parallel momentum SGD that incurs global average across all computing nodes. On the other hand, the large-batch training has been demonstrated critical to achieve runtime speedup. This motivates us to investigate how DmSGD performs in the large-batch scenario.In this work, we find the momentum term can amplify the inconsistency bias in DmSGD. Such bias becomes more evident as batch-size grows large and hence results in severe performance degradation. We next propose Decent-LaM, a novel decentralized large-batch momentum SGD to remove the momentum-incurred bias. The convergence rate for both non-convex and strongly-convex scenarios is established. Our theoretical results justify the superiority of De-centLaM to DmSGD especially in the large-batch scenario. Experimental results on a variety of computer vision tasks and models demonstrate that DecentLaM promises both efficient and high-quality training.

A Stochastic Proximal Gradient Framework for Decentralized Non-Convex Composite Optimization: Topology-Independent Sample Complexity and Communication Efficiency

“…The special case h(x) = 0 of Problem (1) has been relatively well-studied. For this smooth formulation, variants of decentralized stochastic gradient descent (DSGD), e.g., [4,26,52,70], admit simple implementations yet provide competitive practical performance against centralized methods in homogeneous environments like data centers. When the data distributions across the network become heterogeneous, the performance of DSGD in both practice and theory degrades significantly [15,39,57,59,68].…”

“…This cooperative minimization paradigm, built upon local communication and computation, has numerous applications in estimation, control, adaptation, and learning problems that frequently arise in multi-agent systems [8,17,31,57]. In particular, the sparse and localized peer-to-peer information exchange pattern in decentralized networks substantially reduces the communication overhead on the parameter server in the centralized networks, thus making decentralized optimization algorithms especially appealing in large-scale data analytics and machine learning tasks [4,26,70].…”

Exponential Graph is Provably Efficient for Decentralized Deep Training

“…There are many variants of decentralized momentum SGD [3,20,32,67]. This paper will focus on the one proposed by [64] (listed in Algorithm 1), which imposes an additional partialaveraging over the momentum to achieve further speed up.…”

“…In the deep learning regime, decentralize SGD, which was established in [30] to achieve the same linear speedup as parallel SGD in convergence rate, has attracted a lot of attentions. Many efforts have been made to extend the algorithm to directed topologies [3,42], time-varying topologies [25,42], asynchronous settings [31], and data-heterogeneous scenarios [57,62,32,67]. Techniques such as quantization/compression [2,8,26,24,58,36], periodic updates [55,25,64], and lazy communication [37,38,13] were also integrated into decentralized SGD to further reduce communiation overheads.…”

“…In addition to the DmSGD algorithm (Algorithm 1) studied in this paper, we also examine how exponential graphs perform with other commonly-used decentralized momentum method: the vanilla DmSGD [3] which does not exchange momentum between neighbors, and QG-DmSGD [32] which adds a quasi-global momentum to relieve the influence of data heterogeneity. We do not examine DecentLaM [67] and D 2 [57] because both methods require symmetric weight matrix during the training process which exponential graphs cannot provide. We also list the performance of parallel SGD using global averaging as one baseline.…”