Distributed stochastic optimization with gradient tracking over strongly-connected networks

Ran, Xin; Sahu, Anit Kumar; Khan, Usman A.; Kar, Soummya

doi:10.1109/cdc40024.2019.9029217

Cited by 88 publications

(77 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An interesting attribute of some distributed stochastic methods is that they achieve a variance reduction effect similar to mini-batching [25], [41], [43], [46], [50], [53]. However, only a small subset of methods can achieve linear convergence [25], [34], [46], [50]- [52]. In addition, none of these methods is adaptable to varying application conditions and some of them have excessive memory requirements [25], [34] or rely on data reshuffling schemes [51].…”

Section: Introductionmentioning

confidence: 99%

Nested Distributed Gradient Methods with Stochastic Computation Errors

Iakovidou

Wei

2019

2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

View full text Add to dashboard Cite

In this work, we consider the problem of a network of agents collectively minimizing a sum of convex functions. The agents in our setting can only access their local objective functions and exchange information with their immediate neighbors. Motivated by applications where computation is imperfect, including, but not limited to, empirical risk minimization (ERM) and online learning, we assume that only noisy estimates of the local gradients are available. To tackle this problem, we adapt a class of Nested Distributed Gradient methods (NEAR-DGD) to the stochastic gradient setting. These methods have minimal storage requirements, are communication aware and perform well in settings where gradient computation is costly, while communication is relatively inexpensive. We investigate the convergence properties of our method under standard assumptions for stochastic gradients, i.e. unbiasedness and bounded variance. Our analysis indicates that our method converges to a neighborhood of the optimal solution with a linear rate for local strongly convex functions and appropriate constant steplengths. We also show that distributed optimization with stochastic gradients achieves a noise reduction effect similar to mini-batching, which scales favorably with network size. Finally, we present numerical results to demonstrate the effectiveness of our method.

show abstract

Section: Introductionmentioning

confidence: 99%

Nested Distributed Gradient Methods with Stochastic Computation Errors

Iakovidou

Wei

2019

2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

View full text Add to dashboard Cite

show abstract

“…Solutions to the aggregate optimization problem (2) can be pursued by a variety of decentralized algorithms, including primal [1][2][3]8] and primal-dual [9][10][11][12][13] methods. The notion of -differential privacy as a means of quantifying the privacy loss encountered by sharing functions of private data is due to [7,14], as is the Laplace mechanism, which ensures -differential privacy by perturbing the output of the function by Laplacian noise, where the power of the perturbation is calibrated to the sensitivity of the function and the desired privacy level .…”

Section: Related Workmentioning

confidence: 99%

Graph-Homomorphic Perturbations for Private Decentralized Learning

Vlaski

Sayed

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Decentralized algorithms for stochastic optimization and learning rely on the diffusion of information through repeated local exchanges of intermediate estimates. Such structures are particularly appealing in situations where agents may be hesitant to share raw data due to privacy concerns. Nevertheless, in the absence of additional privacy-preserving mechanisms, the exchange of local estimates, which are generated based on private data can allow for the inference of the data itself. The most common mechanism for guaranteeing privacy is the addition of perturbations to local estimates before broadcasting. These perturbations are generally chosen independently at every agent, resulting in a significant performance loss. We propose an alternative scheme, which constructs perturbations according to a particular nullspace condition, allowing them to be invisible (to first order in the step-size) to the network centroid, while preserving privacy guarantees. The analysis allows for general nonconvex loss functions, and is hence applicable to a large number of machine learning and signal processing problems, including deep learning.

show abstract

“…We denote • 2 and ||| • ||| as the Euclidean vector norm and the spectral (matrix) norm, respectively. Since ρ(B − B ∞ ) < 1, it can be shown that there exists a matrix norm ||| • ||| π , formally defined in [20], such that λ := ||| B − B ∞ ||| π < 1.…”

Section: Introduction and Related Workmentioning

confidence: 99%

A Decentralized Variance-Reduced Method for Stochastic Optimization Over Directed Graphs

Qureshi

Ran

Kar

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

In this paper, we propose a decentralized first-order stochastic optimization method Push-SAGA for finite-sum minimization over a strongly connected directed graph. This method features local variance reduction to remove the uncertainty caused by random sampling of the local gradients, global gradient tracking to address the distributed nature of the data, and push-sum consensus to handle the imbalance caused by the directed nature of the underlying graph. We show that, for a sufficiently small step-size, Push-SAGA linearly converges to the optimal solution for smooth and strongly convex problems, making it the first linearly-convergent stochastic algorithm over arbitrary strongly-connected directed graphs. We illustrate the behavior and convergence properties of Push-SAGA with the help of numerical experiments for strongly convex and non-convex problems.

show abstract

Distributed stochastic optimization with gradient tracking over strongly-connected networks

Cited by 88 publications

References 29 publications

Nested Distributed Gradient Methods with Stochastic Computation Errors

Nested Distributed Gradient Methods with Stochastic Computation Errors

Graph-Homomorphic Perturbations for Private Decentralized Learning

A Decentralized Variance-Reduced Method for Stochastic Optimization Over Directed Graphs

Contact Info

Product

Resources

About