Coded Matrix Multiplication on a Group-Based Model

Abstract

Coded distributed computing has been considered as a promising technique which makes large-scale systems robust to the "straggler" workers. Yet, practical system models for distributed computing have not been available that reflect the clustered or grouped structure of real-world computing servers. Neither the large variations in the computing power and bandwidth capabilities across different servers have been properly modeled. We suggest a group-based model to reflect practical conditions and develop an appropriate coding scheme for this model. The suggested code, called group code, employs parallel encoding for each group. We show that the suggested coding scheme can asymptotically achieve optimal computing time in regimes of infinite n, the number of workers. While theoretical analysis is conducted in the asymptotic regime, numerical results also show that the suggested scheme achieves near-optimal computing time for any finite but reasonably large n. Moreover, we demonstrate that the decoding complexity of the suggested scheme is significantly reduced by the virtue of parallel decoding.

On Batch-Processing Based Coded Computing for Heterogeneous Distributed Computing Systems

“…While most CDC schemes consider homogeneous computing nodes, there have been a few recent studies that investigated CDC over heterogeneous computing clusters. In particular, Kim et al [32], [33] considered the matrixvector multiplication problem and presented an optimal load allocation method that achieves a lower bound of the expected latency. Reisizadeh et al [21] introduced a different approach, namely Heterogeneous Coded Matrix Multiplication (HCMM), that can maximize the expected computing results aggregated at the master node.…”

Straggler-Aware Distributed Learning: Communication–Computation Latency Trade-Off

“…Since the following equality holds PS can obtain the full gradient receiving the computation results from all the workers. In contrast to the naive approach, coded computation schemes for distributed matrix multiplication [ 22 , 23 , 32 , 34 ] first encode the submatrices, and then assign them to the workers to achieve a certain tolerance against slow/straggling workers.…”

“…A wealth of straggler avoidance techniques have been proposed in recent years for DGD as well as other distributed computation tasks [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 ]. The common design notion behind all these schemes is the assignment of redundant computations/tasks to workers, such that faster workers can compensate for the stragglers.…”

A Survey of Coded Distributed Computing

“…In practical distributed computing systems, some processing nodes have the same computational capabilities, in terms of the same distributions of computation time, and thus they can be grouped together. By exploiting the group structure and heterogeneities among different groups of processing nodes [141], [142], the implementation of a combination of group codes and an optimal load allocation strategy not only approaches the optimal computation time that is achieved by the MDS codes, but also has low decoding complexity. In addition, by varying the number of allocated rows of the matrix to the workers [142], the computation latency can be reduced by orders of magnitude over the MDS codes with fixed computation load allocation [141] as the number of workers increases.…”

“…By exploiting the group structure and heterogeneities among different groups of processing nodes [141], [142], the implementation of a combination of group codes and an optimal load allocation strategy not only approaches the optimal computation time that is achieved by the MDS codes, but also has low decoding complexity. In addition, by varying the number of allocated rows of the matrix to the workers [142], the computation latency can be reduced by orders of magnitude over the MDS codes with fixed computation load allocation [141] as the number of workers increases. The load allocation strategy proposed in [142] focuses mainly on the design of an optimal MDS code.…”