Compressed Gradient Methods With Hessian-Aided Error Compensation

Khirirat, Sarit; Magnússon, Sindri; Johansson, Mikael

doi:10.1109/tsp.2020.3048229

Cited by 14 publications

(7 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The compressors mentioned above can be unified into three general classes. Specifically, [30] proposed a wider class of compressors with bounded relative compression error which covers the compressors used in [13]- [18]; [31] considered a general class of compressors with globally bounded absolute compression error which accommodates the compressors used in [21]- [23]; and [32] studied a general class of compressors with locally bounded absolute compression error which contains the compressors used in [24]- [29]. These studies also analyzed the convergence properties of the proposed algorithms.…”

Section: A Related Work and Motivationmentioning

confidence: 99%

“…The same class of compressors satisfying Assumption 2 has also been used in [31], which incorporates the deterministic quantization used in [19]- [21] and the unbiased random quantization used in [21], [33], [38].…”

Section: A Compressorsmentioning

confidence: 99%

“…Compared to [31] which used the same class of compressors satisfying Assumption 2, Theorem 6 shows that a global optimum can be precisely found with a linear convergence rate under the P-Ł condition. In contrast, although [31] assumed the stronger strong convexity assumption and also showed that convergence rate is linear, the parallel algorithms proposed in [31] only converged to a neighbor of the unique optimal point.…”

Section: B Convergence Analysismentioning

confidence: 99%

See 2 more Smart Citations

Communication Compression for Distributed Nonconvex Optimization

Yi¹,

Zhang²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper considers decentralized nonconvex optimization with the cost functions being distributed over agents. Noting that information compression is a key tool to reduce the heavy communication load for decentralized algorithms as agents iteratively communicate with neighbors, we propose three decentralized primal-dual algorithms with compressed communication. The first two algorithms are applicable to a general class of compressors with bounded relative compression error and the third algorithm is suitable for two general classes of compressors with bounded absolute compression error.We show that the proposed decentralized algorithms with compressed communication have comparable convergence properties as state-of-the-art algorithms without communication compression. Specifically, we show that they can find first-order stationary points with sublinear convergence rate O(1/T ) when each local cost function is smooth, where T is the total number of iterations, and find global optima with linear convergence rate under an additional condition that the global cost function satisfies the Polyak-Łojasiewicz condition. Numerical simulations are provided to illustrate the effectiveness of the theoretical results.

show abstract

Section: A Related Work and Motivationmentioning

confidence: 99%

Section: A Compressorsmentioning

confidence: 99%

Section: B Convergence Analysismentioning

confidence: 99%

See 1 more Smart Citation

Communication Compression for Distributed Nonconvex Optimization

Yi¹,

Zhang²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Inexact gradient methods are considered in [17]- [25]. These methods are directly related to dual decomposition methods.…”

Section: A Related Workmentioning

confidence: 99%

“…Errors in dual decomposition settings may not fit there, because the assumption can directly impose a requirement on the norm of the true subgradients. References [24], [25] discuss means of modeling inexactness of subgradients again from stochastic/deterministic points of view. They appear to be readily applied in a distributed optimization setting with dual decomposition.…”

Section: A Related Workmentioning

confidence: 99%

On the Primal Feasibility in Dual Decomposition Methods Under Additive and Bounded Errors

Abeynanda¹,

Weeraddana²,

Lanel³

et al. 2021

Preprint

View full text Add to dashboard Cite

With the unprecedented growth of signal processing and machine learning application domains, there has been a tremendous expansion of interest in distributed optimization methods to cope with the underlying large-scale problems. Nonetheless, inevitable system-specific challenges such as limited computational power, limited communication, latency requirements, measurement errors, and noises in wireless channels impose restrictions on the exactness of the underlying algorithms. Such restrictions have appealed to the exploration of algorithms' convergence behaviors under inexact settings. Despite the extensive research conducted in the area, it seems that the analysis of convergences of dual decomposition methods concerning primal optimality violations, together with dual optimality violations is less investigated. Here, we provide a systematic exposition of the convergence of feasible points in dual decomposition methods under inexact settings, for an important class of global consensus optimization problems. Convergences and the rate of convergences of the algorithms are mathematically substantiated, not only from a dual-domain standpoint but also from a primal-domain standpoint. Analytical results show that the algorithms converge to a neighborhood of optimality, the size of which depends on the level of underlying distortions. Further elaboration of a generalized problem formulation is also furnished, together with the convergence properties of the underlying algorithms. Finally, theoretical derivations are verified by numerical experiments.

show abstract

Comparing NARS and Reinforcement Learning: An Analysis of ONA and Q-Learning Algorithms

Beikmohammadi

Magnússon

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Data similarity assumptions have traditionally been relied upon to understand the convergence behaviors of federated learning methods. Unfortunately, this approach often demands fine-tuning step sizes based on the level of data similarity. When data similarity is low, these small step sizes result in an unacceptably slow convergence speed for federated methods. In this paper, we present a novel and unified framework for analyzing the convergence of federated learning algorithms without the need for data similarity conditions. Our analysis centers on an inequality that captures the influence of step sizes on algorithmic convergence performance. By applying our theorems to wellknown federated algorithms, we derive precise expressions for three widely used step size schedules: fixed, diminishing, and step-decay step sizes, which are independent of data similarity conditions. Finally, we conduct comprehensive evaluations of the performance of these federated learning algorithms, employing the proposed step size strategies to train deep neural network models on benchmark datasets under varying data similarity conditions. Our findings demonstrate significant improvements in convergence speed and overall performance, marking a substantial advancement in federated learning research.

show abstract

Compressed Gradient Methods With Hessian-Aided Error Compensation

Cited by 14 publications

References 22 publications

Communication Compression for Distributed Nonconvex Optimization

Communication Compression for Distributed Nonconvex Optimization

On the Primal Feasibility in Dual Decomposition Methods Under Additive and Bounded Errors

Comparing NARS and Reinforcement Learning: An Analysis of ONA and Q-Learning Algorithms

Contact Info

Product

Resources

About