Data-Dependent Convergence for Consensus Stochastic Optimization

Abstract

We study a distributed consensus-based stochastic gradient descent (SGD) algorithm and show that the rate of convergence involves the spectral properties of two matrices: the standard spectral gap of a weight matrix from the network topology and a new term depending on the spectral norm of the sample covariance matrix of the data. This data-dependent convergence rate shows that distributed SGD algorithms perform better on datasets with small spectral norm. Our analysis method also allows us to find data-dependent convergence rates as we limit the amount of communication. Spreading a fixed amount of data across more nodes slows convergence; for asymptotically growing data sets we show that adding more machines can help when minimizing twice-differentiable losses. arXiv:1603.04379v2 [math.OC] 1 Sep 2016In such a computational model, nodes can perform local computations and send messages to each other to jointly minimize (1). The strategy we analyze is what is referred to as distributed primal averaging [3]: each node in the network processes points sequentially, performing a SGD update locally and averaging the current iterate values of their neighbors after each gradient step. This can also be thought of as a distributed consensus-based version of Pegasos [4] when the loss function is the hinge loss. We consider a general topology with m nodes attempting to minimize a global objective function J(w) that decomposes into a sum of m local objectives:. This is a model for optimization in systems such as data centers, distributed control systems, and sensor networks.Main Results. Our goal in this paper is to characterize how the spectral norm ρ 2 = σ 1 (EP [xx ]) of the sample covariance affects the rate of convergence of stochastic consensus schemes under different communication requirements. Elucidating this dependence can help guide empirical practice by providing insight into when these methods will work well. We prove an upper bound on the suboptimality gap for distributed primal averaging that depends on ρ 2 as well as the mixing time of the weight matrix associated to the algorithm. Our result shows that networks of size m < 1 ρ 2 gain from parallelization. To understand the communication-limited regime, we extend our analysis to intermittent communication. In a setting with finite data and sparse connectivity, convergence will deteriorate with increasing m because we split the data to more machines that are farther apart. We also show that by using a mini-batching strategy we can offset the penalty of infrequent communication by communicating after a mini-batch (sub)gradient step. Finally, in an asymptotic regime with infinite data at every node we show using results of Bianchi et al. [5] that for twice-differentiable loss functions this network effect disappears and that we gain from additional parallelization.Related Work. Several authors have proposed distributed algorithms involving nodes computing local gradient steps and averaging iterates, gradients, or other functions of their neighbors [3,6,7]. By alte...

Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms

“…Each node only synchronizes with its neighbors, thus reducing the communication overhead significantly. Decentralized averaging has a long history in the distributed and consensus optimization community (Tsitsiklis et al, 1986;Nedic & Ozdaglar, 2009;Duchi et al, 2012;Tsianos et al, 2012;Zeng & Yin, 2016;Yuan et al, 2016;Sirb & Ye, 2018;Bijral et al, 2017). Most of these works are for gradient descent or dual averaging methods rather than stochastic gradient descent (SGD), and they do not allow workers to make local updates.…”

Scaling-up Distributed Processing of Data Streams for Machine Learning

“…Many papers (e.g., [41], [42]) use the term in the SA sense described here, with a continuous stream of data in which no sample is used more than once. However, other papers (e.g., [2], [43]) use the term within the ERM framework to describe algorithms that operate on a fixed dataset, from which mini-batches of data are sampled with replacement and noisy gradients are computed. To disambiguate, some authors (e.g., [44]) use the term single-pass SGD to indicate the former usage.…”

Balanced Hodge Laplacians Optimize Consensus Dynamics over Simplicial Complexes

“…Notably, consensus dynamics provide a foundation for decentralized optimization algorithms, which strategically implement consensus using synchronous or asynchronous gossiping between nodes 26,27 . Such algorithms are often employed to take advantage of distributed computing infrastructure to more efficiently train machine learning models, such as support vector machines 28,29 and deep neural networks [30][31][32][33] . For such systems, each node trains a local model on local data, and at the same time, communication between nodes enables them to reach a consensus on what the model parameter should be.…”