A sequential approximation framework for coded distributed optimization

Zhu, Jingge; Pu, Ye; Gupta, Vipul; Tomlin, Claire J.; Ramchandran, Kannan

doi:10.1109/allerton.2017.8262879

Cited by 23 publications

(26 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, to provide another estimate with lower error, the input matrices should be newly compressed and a new coding step need to be used for the new random samples of inputs. Another line of work [16] and [17] brings the idea of approximate computing to the encoding and decoding steps. Using maximum distance separable (MDS) codes, [16] and [17] guarantee accuracy improvement over time by intelligently prioritizing and allocating tasks among workers.…”

Section: A Background 1) Polynomial-based Cdcmentioning

confidence: 99%

“…Another line of work [16] and [17] brings the idea of approximate computing to the encoding and decoding steps. Using maximum distance separable (MDS) codes, [16] and [17] guarantee accuracy improvement over time by intelligently prioritizing and allocating tasks among workers. These methods while effective, are applied only to matrix-vector multiplication.…”

Section: A Background 1) Polynomial-based Cdcmentioning

confidence: 99%

See 1 more Smart Citation

Successive Approximation Coding for Distributed Matrix Multiplication

Kiani¹,

Draper²

2022

Preprint

View full text Add to dashboard Cite

Coded distributed computing was recently introduced to mitigate the effect of stragglers on distributed computing. This paper combines ideas of approximate computing with coded computing to further accelerate computation. We propose successive approximation coding (SAC) techniques that realize a tradeoff between accuracy and speed, allowing the distributed computing system to produce approximations that increase in accuracy over time. If a sufficient number of compute nodes finish their tasks, SAC exactly recovers the desired computation. We theoretically provide design guidelines for our SAC techniques, and numerically show that SAC achieves a better accuracy-speed tradeoff in comparison with previous methods.

show abstract

Section: A Background 1) Polynomial-based Cdcmentioning

confidence: 99%

Section: A Background 1) Polynomial-based Cdcmentioning

confidence: 99%

Successive Approximation Coding for Distributed Matrix Multiplication

Kiani¹,

Draper²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The shifted exponential model we considered in above does not reflect this behavior. Thus, k j " n is a possible solution to (9). To robustify the solution to the possible presence of persistent stragglers we change the optimization problem in (9) to max z s.t.…”

Section: Asymptotic Analysismentioning

confidence: 99%

Hierarchical Coded Computation

Ferdinand

Draper

2018

2018 IEEE International Symposium on Information Theory (ISIT)

105

View full text Add to dashboard Cite

Coded computation is a method to mitigate "stragglers" in distributed computing systems through the use of error correction coding that has lately received significant attention. First used in vector-matrix multiplication, the range of application was later extended to include matrix-matrix multiplication, heterogeneous networks, convolution, and approximate computing. A drawback to previous results is they completely ignore work completed by stragglers. While stragglers are slower compute nodes, in many settings the amount of work completed by stragglers can be non-negligible. Thus, in this work, we propose a hierarchical coded computation method that exploits the work completed by all compute nodes. We partition each node's computation into layers of sub-computations such that each layer can be treated as (distinct) erasure channel. We then design different erasure codes for each layer so that all layers have the same failure exponent. We propose design guidelines to optimize parameters of such codes. Numerical results show the proposed scheme has an improvement of a factor of 1.5 in the expected finishing time compared to previous work.

show abstract

“…The second coded computation, the focus of this paper, concept [22] provides resiliency to stragglers and can be utilized to mitigate tail latency in distributed computing [11,22,27,34,38,42]. In particular, several of these works target distributed machine learning.There have been few recent works in the coded computing literature to exploit the computations of slow nodes [41,47], however the key ingredient of our proposed strategy is that it dynamically adapts the computation load of each node to its estimated speed from the previous rounds of computations.…”

Section: Related Workmentioning

confidence: 99%

Slack squeeze coded computing for adaptive straggler mitigation

Narra

Lin

Kiamari

et al. 2019

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

While performing distributed computations in today's cloud-based platforms, execution speed variations among compute nodes can significantly reduce the performance and create bottlenecks like stragglers. Coded computation techniques leverage coding theory to inject computational redundancy and mitigate stragglers in distributed computations. In this paper, we propose a dynamic workload distribution strategy for coded computation called Slack Squeeze Coded Computation (S 2 C 2 ). S 2 C 2 squeezes the compute slack (i.e., overhead) that is built into the coded computing frameworks by efficiently assigning work for all fast and slow nodes according to their speeds and without needing to re-distribute data. We implement an LSTM-based speed prediction algorithm to predict speeds of compute nodes. We evaluate S 2 C 2 on linear algebraic algorithms, gradient descent, graph ranking, and graph filtering algorithms. We demonstrate 19% to 39% reduction in total computation latency using S 2 C 2 compared to job replication and coded computation. We further show how S 2 C 2 can be applied beyond matrix-vector multiplication.

show abstract

A sequential approximation framework for coded distributed optimization

Cited by 23 publications

References 12 publications

Successive Approximation Coding for Distributed Matrix Multiplication

Successive Approximation Coding for Distributed Matrix Multiplication

Hierarchical Coded Computation

Slack squeeze coded computing for adaptive straggler mitigation

Contact Info

Product

Resources

About