C³LES: Codes for Coded Computation that Leverage Stragglers

Abstract

In distributed computing systems, it is well recognized that worker nodes that are slow (called stragglers) tend to dominate the overall job execution time. Coded computation utilizes concepts from erasure coding to mitigate the effect of stragglers by running "coded" copies of tasks comprising a job. Stragglers are typically treated as erasures in this process.While this is useful, there are issues with applying, e.g., MDS codes in a straightforward manner. Specifically, several applications such as matrix-vector products deal with sparse matrices. MDS codes typically require dense linear combinations of submatrices of the original matrix which destroy their inherent sparsity. This is problematic as it results in significantly higher processing times for computing the submatrix-vector products in coded computation. Furthermore, it also ignores partial computations at stragglers.In this work, we propose a fine-grained model that quantifies the level of non-trivial coding needed to obtain the benefits of coding in matrix-vector computation. Simultaneously, it allows us to leverage partial computations performed by the straggler nodes. For this model, we propose and evaluate several code designs and discuss their properties.

Straggler-Aware Distributed Learning: Communication–Computation Latency Trade-Off

“…We note that in ( 6 ) also remains unchanged throughout GD iterations. Hence, if can be computed at the beginning, the main computational task reduces to linear operations at each iteration, which allows employing various linear coding structures, e.g., maximum distance separable (MDS) codes, or rateless codes, to encode rows of to achieve robustness against stragglers [ 17 , 18 , 19 , 25 , 28 , 29 ].…”

“…A wealth of straggler avoidance techniques have been proposed in recent years for DGD as well as other distributed computation tasks [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 ]. The common design notion behind all these schemes is the assignment of redundant computations/tasks to workers, such that faster workers can compensate for the stragglers.…”

Securely Straggler-Exploiting Coded Computation for Distributed Matrix Multiplication

“…The authors in [13] have introduced a new task allocation scheme to assign multiple small sub-tasks to the workers for a straggler-exploiting scenario, instead of assigning a single big sub-task to each worker. In addition, the authors in [14] have proposed a new fine-grained task allocation scheme to efficiently utilize the straggling workers' computation ability to speed up the entire task execution. Furthermore, recent papers [15]- [17] have suggested new encoding schemes for matrix multiplication to efficiently utilize the memory of the workers and to lower the communication load in distributed computing.…”

Squeezed Polynomial Codes: Communication-Efficient Coded Computation in Straggler-Exploiting Distributed Matrix Multiplication

“…In addition, the authors in [16] have suggested a bi-variate encoding scheme for matrix-matrix multiplication in a straggler-exploiting scenario to efficiently utilize the memory of the workers and to lower the communication load from a master to its workers compared to the task allocation scheme in [15], under the more strict recovery condition at the master. Furthermore, a straggler-exploiting coded computation for matrix multiplication has been explored different distributed computing environments such as privacy and security constraints [17]- [19].…”