Approximate Weighted C R Coded Matrix Multiplication

Abstract

One of the most common operations in signal processing is matrix multiplication. However, it presents a major computational bottleneck when the matrix dimension is high, as can occur for large data size or feature dimension. Two different approaches to overcoming this bottleneck are: 1) low rank approximation of the matrix product; and 2) distributed computation. We propose a scheme that combines these two approaches. To enable distributed low rank approximation, we generalize the approximate matrix CR-multiplication to accommodate weighted block sampling, and we introduce a weighted coded matrix multiplication method. This results in novel approximate weighted CR coded matrix multiplication schemes, which achieve improved performance for distributed matrix multiplication and are robust to stragglers.

Orthonormal Sketches for Secure Coded Regression

“…There is a benefit when applying a garbled Hadamard transform in this scenario, as the complexity of multiplication resulting from the sketching is less than that of regular multiplication. Also, if such a random projection is used before performing CR-multiplication distributively [15], [16], [21], the approximate result will be the same. Moreover, our dimensionality reduction algorithm can be utilized by a single server, to store low-rank approximations of very large data matrices.…”

Weighted Gradient Coding with Leverage Score Sampling

“…The paper titled "Anytime Coding for Distributed Computation" [47] proposes replicating subtasks according to the job, while [33] and [48] incorporate sketching into CC. It is worth noting that even though we focus on gradient methods in this paper; our approach also applies to second-order methods, as well as approximate matrix products through the CR-multiplication algorithm [9], [28], [49]. We briefly discuss this in Section V.…”

“…This framework accommodates a central class of sketching algorithms, that of importance (block) sampling algorithms (e.g. CU R decomposition [27], CR-multiplication [28]). Coded computing is a novel computing paradigm that utilizes coding theory to effectively inject and leverage data and computation redundancy to mitigate errors and slow or non-responsive servers; known as stragglers, among other fundamental bottlenecks, in large-scale distributed computing.…”

“…More recent works deal with recovering good approximations, while some have utilized techniques from RandNLA; e.g. [28], [30]- [34]. Our results are presented in terms of the standard CC probabilistic model proposed in [29], though they extend to any computing network comprised of a central server and computing nodes, referred to as servers.…”

“…The tasks' computation times follow a runtime distribution [29], which along with a prespecified gradient transmission "ending time" T , determine the number of replications per task across the network. Though similar ideas have been proposed [28], [31]- [33]; where sketching has been incorporated into CC, this is the first time redundancy is introduced through RandNLA; as opposed to compression, to obtain approximation guarantees. In terms of CC, though uniform replication is a very powerful technique, it does not capture the relevance between the information of the dataset.…”