D-FW: Communication efficient distributed algorithms for high-dimensional sparse optimization

Lafond, Jean; Wai, Hoi-To; Moulines, Éric

doi:10.1109/icassp.2016.7472457

Cited by 10 publications

(16 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that the bounded gradient assumption is a regular assumption in the convergence analysis of decentralized gradient methods (see, [4], [5], [18], [35], [36], [56], [57], [27], [62] for example), even in the convex setting [24] and also [10], though it is not required for centralized gradient descent.…”

Section: B Convergence Results Of Dgdmentioning

confidence: 99%

“…In [62], the authors established convergence results similar to Prox-DGD under diminishing step sizes. The stochastic version of DeFW has also been developed in [27] for high-dimensional convex sparse optimization. The next one is projected stochastic gradient algorithm (Proj SGD) [4] for constrained, nonconvex, smooth consensus optimization with a convex constrained set.…”

Section: Related Work and Discussionmentioning

confidence: 99%

“…When f i 's are convex, the existing algorithms include the (sub)gradient methods [8], [10], [24], [37], [40], [59], [65], [46], and the primal-dual domain methods such as the decentralized alternating direction method of multipliers (DADMM) [51], [52], [9], DLM [31], and EXTRA [53], [54]. When f i 's are nonconvex, some existing results include [4], [5], [18], [35], [36], [56], [57], [27], [60], [62], [68]. In spite of the algorithms and their analysis in these works, the convergence of the simple algorithm Decentralized Gradient Descent (DGD) [40] under nonconvex f i 's is still unknown.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

On Nonconvex Decentralized Gradient Descent

Zeng

Yin

2018

IEEE Trans. Signal Process.

158

107

View full text Add to dashboard Cite

Consensus optimization has received considerable attention in recent years. A number of decentralized algorithms have been proposed for convex consensus optimization. However, to the behaviors or consensus nonconvex optimization, our understanding is more limited.When we lose convexity, we cannot hope our algorithms always return global solutions though they sometimes still do. Somewhat surprisingly, the decentralized consensus algorithms, DGD and Prox-DGD, retain most other properties that are known in the convex setting. In particular, when diminishing (or constant) step sizes are used, we can prove convergence to a (or a neighborhood of) consensus stationary solution under some regular assumptions. It is worth noting that Prox-DGD can handle nonconvex nonsmooth functions if their proximal operators can be computed. Such functions include SCAD, MCP and ℓq quasi-norms, q ∈ [0, 1). Similarly, Prox-DGD can take the constraint to a nonconvex set with an easy projection.To establish these properties, we have to introduce a completely different line of analysis, as well as modify existing proofs that were used in the convex setting.

show abstract

Section: B Convergence Results Of Dgdmentioning

confidence: 99%

Section: Related Work and Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

On Nonconvex Decentralized Gradient Descent

Zeng

Yin

2018

IEEE Trans. Signal Process.

158

107

View full text Add to dashboard Cite

show abstract

“…In the constrained setting, and in particular for distributed FW algorithms, the communicationefficient versions were only studied for specific problems such as sparse learning (Bellet et al, 2015;Lafond et al, 2016). In this paper, however, we develop Quantized Frank-Wolfe (QFW), a general communication-efficient distributed FW for both convex and non-convex objective functions.…”

Section: Introductionmentioning

confidence: 99%

Quantized Frank-Wolfe: Faster Optimization, Lower Communication, and Projection Free

Zhang,

Chen,

Mokhtari

et al. 2019

Preprint

View full text Add to dashboard Cite

How can we efficiently mitigate the overhead of gradient communications in distributed optimization? This problem is at the heart of training scalable machine learning models and has been mainly studied in the unconstrained setting. In this paper, we propose Quantized Frank-Wolfe (QFW), the first projection-free and communication-efficient algorithm for solving constrained optimization problems at scale. We consider both convex and non-convex objective functions, expressed as a finite-sum or more generally a stochastic optimization problem, and provide strong theoretical guarantees on the convergence rate of QFW. This is accomplished by proposing novel quantization schemes that efficiently compress gradients while controlling the noise variance introduced during this process. Finally, we empirically validate the efficiency of QFW in terms of communication and the quality of returned solution against natural baselines.

show abstract

“…for all t ≥ 1, where θ is an optimal solution to (1). Furthermore, if F is µ-strongly convex and the optimal solution θ lies in the interior of C, i.e., δ > 0 (cf.…”

mentioning

confidence: 99%

Decentralized Frank–Wolfe Algorithm for Convex and Nonconvex Problems

Wai

Lafond

Scaglione

et al. 2017

IEEE Trans. Automat. Contr.

Self Cite

View full text Add to dashboard Cite

Decentralized optimization algorithms have received much attention due to the recent advances in network information processing. However, conventional decentralized algorithms based on projected gradient descent are incapable of handling high dimensional constrained problems, as the projection step becomes computationally prohibitive to compute. To address this problem, this paper adopts a projection-free optimization approach, a.k.a. the Frank-Wolfe (FW) or conditional gradient algorithm. We first develop a decentralized FW (DeFW) algorithm from the classical FW algorithm. The convergence of the proposed algorithm is studied by viewing the decentralized algorithm as an inexact FW algorithm. Using a diminishing step size rule and letting t be the iteration number, we show that the DeFW algorithm's convergence rate is O(1/t) for convex objectives; is O(1/t 2 ) for strongly convex objectives with the optimal solution in the interior of the constraint set; and is O(1/ √ t) towards a stationary point for smooth but non-convex objectives. We then show that a consensus-based DeFW algorithm meets the above guarantees with two communication rounds per iteration. Furthermore, we demonstrate the advantages of the proposed DeFW algorithm on low-complexity robust matrix completion and communication efficient sparse learning. Numerical results on synthetic and real data are presented to support our findings.

show abstract

D-FW: Communication efficient distributed algorithms for high-dimensional sparse optimization

Cited by 10 publications

References 15 publications

On Nonconvex Decentralized Gradient Descent

On Nonconvex Decentralized Gradient Descent

Quantized Frank-Wolfe: Faster Optimization, Lower Communication, and Projection Free

Decentralized Frank–Wolfe Algorithm for Convex and Nonconvex Problems

Contact Info

Product

Resources

About