Weighted Gradient Coding with Leverage Score Sampling

Charalambides, Neophytos; Pilancı, Mert; Hero, Alfred O.

doi:10.1109/icassp40776.2020.9054153

Cited by 9 publications

(14 citation statements)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One could sample a fixed number of times, as is done in most such algorithms. We describe the case where t distinct pairs are sampled, to make the connection with distributed computations more natural; as was done in [13].…”

Section: Weighted Cr-multiplicationmentioning

confidence: 99%

Approximate Weighted C R Coded Matrix Multiplication

Charalambides

Pilancı

Hero

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

One of the most common operations in signal processing is matrix multiplication. However, it presents a major computational bottleneck when the matrix dimension is high, as can occur for large data size or feature dimension. Two different approaches to overcoming this bottleneck are: 1) low rank approximation of the matrix product; and 2) distributed computation. We propose a scheme that combines these two approaches. To enable distributed low rank approximation, we generalize the approximate matrix CR-multiplication to accommodate weighted block sampling, and we introduce a weighted coded matrix multiplication method. This results in novel approximate weighted CR coded matrix multiplication schemes, which achieve improved performance for distributed matrix multiplication and are robust to stragglers.

show abstract

Section: Weighted Cr-multiplicationmentioning

confidence: 99%

Approximate Weighted C R Coded Matrix Multiplication

Charalambides

Pilancı

Hero

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…These schemes achieve a better threshold, as they encode linear combinations of submatrices of one or both the matrices, and then carry out the computation on the encoded submatrices, from which a fraction of all the assigned tasks they then decode to retrieve the matrix product. As in GC, in our CMMSs we first carry out the computations and then encode them locally at the worker nodes, e.g., (25) and (28). Our CMMSs meet the optimal recovery threshold known for GC, as they are derived from a GCS.…”

Section: Comparison To Prior Workmentioning

confidence: 99%

“…The Reed-Solomon based scheme was also used as a basis for weighted gradient coding [28]. The idea behind the weighting is similar to that of the weighted CMMS [48] described in V-E.…”

Section: A Reed-solomon Scheme and Weighted Gradient Codingmentioning

confidence: 99%

“…The problem of exact recovery of the gradient was studied in several prior works, e.g., [14]- [18], while the numerical stability issue was studied in [19]. There are also several works involving GC for approximate recovery of the gradient [16], [20]- [28]. Also, an idea similar to GC had appeared in [29], though not within a coding-theoretic setting.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Numerically Stable Binary Gradient Coding

Charalambides

Mahdavifar

Hero

2020

2020 IEEE International Symposium on Information Theory (ISIT)

Self Cite

View full text Add to dashboard Cite

This paper addresses the gradient coding and coded matrix multiplication problems in distributed optimization and coded computing. We present a numerically stable binary coding method which overcomes the drawbacks of the gradient coding method proposed by Tandon et al., and can also be leveraged by coded computing networks whose servers are of heterogeneous nature. The proposed binary encoding avoids operations over the real and complex numbers which are inherently numerically unstable, thereby enabling numerically stable distributed encodings of the partial gradients. We then make connections between gradient coding and coded matrix multiplication. Specifically, we show that any gradient coding scheme can be extended to coded matrix multiplication. Furthermore, we show how the proposed binary gradient coding scheme can be used to construct three different coded matrix multiplication schemes, each achieving different trade-offs.

show abstract

“…The gradient coding (GC) scheme, introduced in [8], considers gradient estimates on subsets of a dataset as partial computations, and achieves redundancy by replicating parts of the dataset at multiple workers. This approach has been extended in various directions to improve the performance [9]- [14] Let G = {g 1 , . .…”

Section: Extension To Coded Communication Scenariomentioning

confidence: 99%

Coded Distributed Computing with Partial Recovery

Ozfatura¹,

Ulukuş²,

Gündüz³

2020

Preprint

View full text Add to dashboard Cite

Coded computation techniques provide robustness against straggling workers in distributed computing. However, most of the existing schemes require exact provisioning of the straggling behaviour and ignore the computations carried out by straggling workers. Moreover, these schemes are typically designed to recover the desired computation results accurately, while in many machine learning and iterative optimization algorithms, faster approximate solutions are known to result in an improvement in the overall convergence time. In this paper, we first introduce a novel coded matrix-vector multiplication scheme, called coded computation with partial recovery (CCPR), which benefits from the advantages of both coded and uncoded computation schemes, and reduces both the computation time and the decoding complexity by allowing a trade-off between the accuracy and the speed of computation. We then extend this approach to distributed implementation of more general computation tasks by proposing a coded communication scheme with partial recovery, where the results of subtasks computed by the workers are coded before being communicated. Numerical simulations on a large linear regression task confirm the benefits of the proposed distributed computation scheme with partial recovery in terms of the trade-off between the computation accuracy and latency.

show abstract

Weighted Gradient Coding with Leverage Score Sampling

Cited by 9 publications

References 70 publications

Approximate Weighted C R Coded Matrix Multiplication

Approximate Weighted C R Coded Matrix Multiplication

Numerically Stable Binary Gradient Coding

Coded Distributed Computing with Partial Recovery

Contact Info

Product

Resources

About