Coordinate Descent with Bandit Sampling

Salehi, Farnood; Thiran, Patrick; Celis, L. Elisa

doi:10.48550/arxiv.1712.03010

Search citation statements

Order By: Relevance

Paper Sections

Select...

Introduction1

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2018

Publication Types

Select...

Other1

Relationship

Self Cite0

Independent1

Authors

Journals

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In large-scale machine learning, stochastic optimization algorithms are very popular, see e.g. the parallel stochastic algorithms of [13,29]. Such algorithms are highly iterative and then require low-latency and high-throughput connections.…”

Section: Introductionmentioning

confidence: 99%

Distributed Learning with Sparse Communications by Identification

Grishchenko¹,

Iutzeler²,

Malick³

et al. 2018

Preprint

View full text Add to dashboard Cite

In this paper, we present an asynchronous optimization algorithm for distributed learning, that efficiently reduces the communications between a master and working machines by randomly sparsifying the local updates. This sparsification allows to lift the communication bottleneck often present in distributed learning setups where computations are performed by workers on local data while a master machine coordinates their updates to optimize a global loss.We prove that despite its sparse asynchronous communications, our algorithm allows for a fixed stepsize and benefits from a linear convergence rate in the strongly convex case. Moreover, for 1-regularized problems, this algorithm identifies near-optimal sparsity patterns, so that all communications eventually become sparse. We furthermore leverage on this identification to improve our sparsification technique. We illustrate on real and synthetic data that this algorithm converges faster in terms of data exchanges.

show abstract

Section: Introductionmentioning

confidence: 99%