Multi-consensus Decentralized Accelerated Gradient Descent

Abstract

This paper considers the decentralized optimization problem, which has applications in large scale machine learning, sensor networks, and control theory. We propose a novel algorithm that can achieve near optimal communication complexity, matching the known lower bound up to a logarithmic factor of the condition number of the problem. Our theoretical results give affirmative answers to the open problem on whether there exists an algorithm that can achieve a communication complexity (nearly) matching the lower bound depending on the global condition number instead of the local one. Moreover, the proposed algorithm achieves the optimal computation complexity matching the lower bound up to universal constants. Furthermore, to achieve a linear convergence rate, our algorithm doesn't require the individual functions to be (strongly) convex. Our method relies on a novel combination of known techniques including Nesterov's accelerated gradient descent, multi-consensus and gradient-tracking. The analysis is new, and may be applied to other related problems. Empirical studies demonstrate the effectiveness of our method for machine learning applications.

Acceleration in Distributed Optimization under Similarity

“…• Acceleration over mesh networks: Given the focus of this work, we comment next only distributed algorithms over mesh networks employing some form of acceleration and provably convergent-they are summarized in Table 1. Although substantially different-some are primal [Ye et al, 2020a, Ye et al, 2020b, Li and Lin, 2020, Rogozin et al, 2020 others are dual or penalty-based [Scaman et al, 2017, Uribe et al, 2020, Li et al, 2018 methods, and applicable to special instances of (P) (mainly with r = 0) and subject to special design constraints (e.g., positive semidefinite gossip matrix)-they all achieve linear convergence rate, with communication complexity scaling some with √ κ ℓ (κ ℓ = L mx /µ mn is the "local" condition number) and others with √ κ (κ = L/µ is the condition number of f ). Note that in general κ ≪ κ ℓ ; hence the latter group is preferable to the former.…”

“…OPAPC , Accelerated Dual Ascent [Uribe et al, 2020, Alg. 3], APM-C [Li et al, 2018], Mudag [Ye et al, 2020a], Accelerated EXTRA [Li and Lin, 2020], DAccGD [Rogozin et al, 2020], and DPAG [Ye et al, 2020b]. L (resp.…”

Optimal Gradient Tracking for Decentralized Optimization

“…It was shown in [46] that to obtain ǫ-optimal solutions, the gradient computation complexity is lower bounded by O √ κ log 1 ǫ , and the communication complexity is lower bounded by O κ θ log 1 ǫ . To obtain better complexities, many accelerated decentralized gradient-type methods have been developed (e.g., [11,12,16,18,20,21,22,23,24,38,41,42,43,46,54,57,61,62]). There exist dual-based methods such as [46] that achieve optimal complexities.…”

“…In this paper, we focus on dual-free methods or gradient-type methods only. Some algorithms, for instance [16,22,23,41,42,61,62], rely on inner loops to guarantee desirable convergence rates. However, inner loops place a larger communication burden [24,38] which may limit the applications of these methods, since communication has often been recognized as the major bottleneck in distributed or decentralized optimization.…”

“…Gradient tracking (GT) is one of the most popular techniques used for developing accelerated decentralized optimization methods. Most of the existing GT-based accelerated methods rely on inner loops of multiple consensus steps or Chebyshev acceleration (CA) to reduce the consensus errors (the difference among the local variables of different agents) between consecutive steps of the outer loop; see for instance [16,22,23,41,42,61,62]. Among single-loop methods, Acc-DNGD-SC [38] achieves O κ 5/7 θ 3/2 log 1 ǫ gradient computation and communication complexities.…”