Optimization-based Block Coordinate Gradient Coding

Wang, Qi; Cui, Ying; Li, Chenglin; Xiong, Hongkai

doi:10.1109/globecom46510.2021.9685858

Cited by 3 publications

(7 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The master aims to minimize the risk function with respect to the model parameter θ using commonly used gradient descent methods. 1 To handle a massive amount of training data, the master implements a gradient descent method with the help of all workers. Specifically, the master partitions the whole data set and sends to each worker some particular subsets so that the master and N workers can collaboratively compute the gradient ∇ θ ˆ (θ; D) y∈D ∇ θ (θ; y) in each iteration.…”

Section: System Settingmentioning

confidence: 99%

“…We would like to minimize E [τ (s, T)] by optimizing the coding parameters s for the L coordinates under the constraints in (1).…”

Section: A Problem Formulationmentioning

confidence: 99%

“…Proof: Please refer to Appendix A. Lemma 1, together with the constraints in (1), indicates that we can optimally partition the L coordinates into N blocks, each with identical redundancy, for tolerating 0, 1, • • • , N − 1 stragglers, respectively. That is, the proposed coordinate gradient coding scheme with s * becomes a block coordinate gradient coding scheme with at most N blocks, each with one coding parameter.…”

Section: A Problem Formulationmentioning

confidence: 99%

“…The authors are with Shanghai Jiao Tong University, China. This paper was presented in part at IEEE GLOBECOM 2021 [1]. such a distributed computation system, due to various factors such as insufficient power, contention of shared resources, imbalanced work allocation and network congestions [2], [3], some processing nodes may be slower than others or even fail from time to time.…”

Section: Introductionmentioning

confidence: 99%

“…To the best of our knowledge, this is the first work that optimizes the redundancies for gradient coding to effectively utilize the computing capabilities of partial stragglers. The preliminary version of this paper appeared as [1]. In this paper, we additionally consider the minimization of the completion probability consisting of theoretical analysis and numerical results.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Optimization-based Block Coordinate Gradient Coding for Mitigating Partial Stragglers in Distributed Learning

Wang¹,

Cui²,

Li³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Gradient coding schemes effectively mitigate full stragglers in distributed learning by introducing identical redundancy in coded local partial derivatives corresponding to all model parameters. However, they are no longer effective for partial stragglers as they cannot utilize incomplete computation results from partial stragglers. This paper aims to design a new gradient coding scheme for mitigating partial stragglers in distributed learning. Specifically, we consider a distributed system consisting of one master and N workers, characterized by a general partial straggler model and focuses on solving a general large-scale machine learning problem with L model parameters using gradient coding. First, we propose a coordinate gradient coding scheme with L coding parameters representing L possibly different diversities for the L coordinates, which generates most gradient coding schemes. Then, we consider the minimization of the expected overall runtime and the maximization of the completion probability with respect to the L coding parameters for coordinates, which are challenging discrete optimization problems. To reduce computational complexity, we first transform each to an equivalent but much simpler discrete problem with N L variables representing the partition of the L coordinates into N blocks, each with identical redundancy. This indicates an equivalent but more easily implemented block coordinate gradient coding scheme with N coding parameters for blocks. Then, we adopt continuous relaxation to further reduce computational complexity. For the resulting minimization of expected overall runtime, we develop an iterative algorithm of computational complexity O(N 2 ) to obtain an optimal solution and derive two closed-form approximate solutions both with computational complexity O(N ). For the resultant maximization of the completion probability, we develop an iterative algorithm of computational complexity O(N 2 ) to obtain a stationary point and derive a closed-form approximate solution with computational complexity O(N ) at a large threshold. Finally, numerical results show that the proposed solutions significantly outperform existing coded computation schemes and their extensions.

show abstract

Section: System Settingmentioning

confidence: 99%

“…We would like to minimize E [τ (s, T)] by optimizing the coding parameters s for the L coordinates under the constraints in (1).…”

Section: A Problem Formulationmentioning

confidence: 99%

Section: A Problem Formulationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Optimization-based Block Coordinate Gradient Coding for Mitigating Partial Stragglers in Distributed Learning

Wang¹,

Cui²,

Li³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Randomized block-coordinate adaptive algorithms for nonconvex optimization problems

Zhou

Huang

et al. 2023

Engineering Applications of Artificial Intelligence

View full text Add to dashboard Cite

Optimization-Based Block Coordinate Gradient Coding for Mitigating Partial Stragglers in Distributed Learning

Wang

Cui

et al. 2023

IEEE Trans. Signal Process.

Self Cite

View full text Add to dashboard Cite

Edge computing has recently emerged as a promising paradigm to boost the performance of distributed learning by leveraging the distributed resources at edge nodes. Architecturally, the introduction of edge nodes adds an additional intermediate layer between the master and workers in the original distributed learning systems, potentially leading to more severe straggler effect. Recently, coding theory-based approaches have been proposed for stragglers mitigation in distributed learning, but the majority focus on the conventional workers-master architecture. In this paper, along a different line, we investigate the problem of mitigating the straggler effect in hierarchical distributed learning systems with an additional layer composed of edge nodes. Technically, we first derive the fundamental trade-off between the computational loads of workers and the stragglers tolerance. Then, we propose a hierarchical gradient coding framework, which provides better stragglers mitigation, to achieve the derived computational trade-off. To further improve the performance of our framework in heterogeneous scenarios, we formulate an optimization problem with the objective of minimizing the expected execution time for each iteration in the learning process. We develop an efficient algorithm to mathematically solve the problem by outputting the optimum strategy. Extensive simulation results demonstrate the superiority of our schemes compared with conventional solutions.

show abstract

Optimization-based Block Coordinate Gradient Coding

Cited by 3 publications

References 10 publications

Optimization-based Block Coordinate Gradient Coding for Mitigating Partial Stragglers in Distributed Learning

Optimization-based Block Coordinate Gradient Coding for Mitigating Partial Stragglers in Distributed Learning

Randomized block-coordinate adaptive algorithms for nonconvex optimization problems

Optimization-Based Block Coordinate Gradient Coding for Mitigating Partial Stragglers in Distributed Learning

Contact Info

Product

Resources

About