A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!

Abstract

Decentralized optimization methods enable on-device training of machine learning models without a central coordinator. In many scenarios communication between devices is energy demanding and time consuming and forms the bottleneck of the entire system. We propose a new randomized first-order method which tackles the communication bottleneck by applying randomized compression operators to the communicated messages. By combining our scheme with a new variance reduction technique that progressively throughout the iterations reduces the adverse effect of the injected quantization noise, we obtain the first scheme that converges linearly on strongly convex decentralized problems while using compressed communication only. We prove that our method can solve the problems without any increase in the number of communications compared to the baseline which does not perform any communication compression while still allowing for a significant compression factor which depends on the conditioning of the problem and the topology of the network. Our key theoretical findings are supported by numerical experiments.

Quantized Distributed Gradient Tracking Algorithm with Linear Convergence in Directed Networks

“…Recalling the definition η = π T A πBη, then the above inequality can be rewritten as Γ1η 2 + Γ2η − Γ3 < 0 with Γ1, Γ2 and Γ3 defined in (17). Hence, it can be derived that η ≤…”

“…directed networks linear convergence binary quantizer [12], [14] [13], [15] [17]- [21] [22], [23] [2], [24], [25] Our work between the convergence speed and the communication cost per iteration so that linear convergence can be guaranteed by only a few bits quantization. Although the aforementioned quantized algorithms [17]- [19] converge linearly, they are designated only for undirected networks. Note that extending distributed algorithms from undirected networks to directed networks is non-trivial [22]- [25].…”

“…Let h(k) = Cξ k , where C is a positive constant, and ξ ∈ (ρ, 1). The step size η is chosen according to (17). Then the quantizers will never saturate provided that the quantization levels satisfy the following update rule:…”

BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify Communication-Efficient Federated Learning

“…However, the analysis relies on the assumption of bounded stochastic gradients. Free of this assumption, gradient difference compression is also provably able to reduce the compression noise, requiring the use of unbiased compressors [22,23,28,29]. Nevertheless, the influence of gradient difference compression on the Byzantine-robustness has not yet been investigated.…”

“…Variance reduction techniques have been widely used to reduce stochastic noise to accelerate convergence of stochastic algorithms [32,33,21]. In [23,28], the combination of variance reduction and gradient difference compression is investigated. Variance reduction is also important to Byzantine-robustness.…”