Irregular Product Coded Computation for High-Dimensional Matrix Multiplication

Park, Hyegyeong; Moon, Jaekyun

doi:10.1109/isit.2019.8849236

Cited by 12 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the linear regime, the one-dimensional decoding of the MDS-coded schemes is sufficient to recover the missing entries of the computation results. By allowing each row and column of the MDS constituent codes to have different code rates [92], the average computation time can be further reduced, contributing to a decrease in the overall job execution time. Product codes can also be used to solve higher-dimensional linear operations such as tensor operations by exploiting the tensor-structured encoding matrix [93].…”

Section: B Cdc To Mitigate the Straggler Effectsmentioning

confidence: 99%

A Survey of Coded Distributed Computing

Ng¹,

Lim²,

Luong³

et al. 2020

Preprint

View full text Add to dashboard Cite

Distributed computing has become a common approach for large-scale computation of tasks due to benefits such as high reliability, scalability, computation speed, and costeffectiveness. However, distributed computing faces critical issues related to communication load and straggler effects. In particular, computing nodes need to exchange intermediate results with each other in order to calculate the final result, and this significantly increases communication overheads. Furthermore, a distributed computing network may include straggling nodes that run intermittently slower. This results in a longer overall time needed to execute the computation tasks, thereby limiting the performance of distributed computing. To address these issues, coded distributed computing (CDC), i.e., a combination of coding theoretic techniques and distributed computing, has been recently proposed as a promising solution. Coding theoretic techniques have proved effective in WiFi and cellular systems to deal with channel noise. Therefore, CDC may significantly reduce communication load, alleviate the effects of stragglers, provide fault-tolerance, privacy and security. In this survey, we first introduce the fundamentals of CDC, followed by basic CDC schemes. Then, we review and analyze a number of CDC approaches proposed to reduce the communication costs, mitigate the straggler effects, and guarantee privacy and security. Furthermore, we present and discuss applications of CDC in modern computer networks. Finally, we highlight important challenges and promising research directions related to CDC.

show abstract

Section: B Cdc To Mitigate the Straggler Effectsmentioning

confidence: 99%

A Survey of Coded Distributed Computing

Ng¹,

Lim²,

Luong³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…A wealth of straggler avoidance techniques have been proposed in recent years for DGD as well as other distributed computation tasks [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 ]. The common design notion behind all these schemes is the assignment of redundant computations/tasks to workers, such that faster workers can compensate for the stragglers.…”

Section: Introductionmentioning

confidence: 99%

Straggler-Aware Distributed Learning: Communication–Computation Latency Trade-Off

Ozfatura

Ulukuş

Gündüz

2020

Entropy

View full text Add to dashboard Cite

When gradient descent (GD) is scaled to many parallel workers for large-scale machine learning applications, its per-iteration computation time is limited by straggling workers. Straggling workers can be tolerated by assigning redundant computations and/or coding across data and computations, but in most existing schemes, each non-straggling worker transmits one message per iteration to the parameter server (PS) after completing all its computations. Imposing such a limitation results in two drawbacks: over-computation due to inaccurate prediction of the straggling behavior, and under-utilization due to discarding partial computations carried out by stragglers. To overcome these drawbacks, we consider multi-message communication (MMC) by allowing multiple computations to be conveyed from each worker per iteration, and propose novel straggler avoidance techniques for both coded computation and coded communication with MMC. We analyze how the proposed designs can be employed efficiently to seek a balance between the computation and communication latency. Furthermore, we identify the advantages and disadvantages of these designs in different settings through extensive simulations, both model-based and real implementation on Amazon EC2 servers, and demonstrate that proposed schemes with MMC can help improve upon existing straggler avoidance schemes.

show abstract

“…We start the second phase of the proposed worker placement algorithm to place w 12 into a cluster that has a worker which can be assigned to the first cluster. We see from (18) that w 12 can be assigned to clusters 3 or 4. None of the workers which has been assigned to cluster 3 in Phase I can be assigned to the first cluster.…”

mentioning

confidence: 96%

Gradient Coding with Dynamic Clustering for Straggler-Tolerant Distributed Learning

Buyukates¹,

Ozfatura²,

Ulukuş³

et al. 2021

Preprint

View full text Add to dashboard Cite

Distributed implementations are crucial in speeding up large scale machine learning applications.Distributed gradient descent (GD) is widely employed to parallelize the learning task by distributing the dataset across multiple workers. A significant performance bottleneck for the per-iteration completion time in distributed synchronous GD is straggling workers. Coded distributed computation techniques have been introduced recently to mitigate stragglers and to speed up GD iterations by assigning redundant computations to workers. In this paper, we consider gradient coding (GC), and propose a novel dynamic GC scheme, which assigns redundant data to workers to acquire the flexibility to dynamically choose from among a set of possible codes depending on the past straggling behavior. In particular, we consider GC with clustering, and regulate the number of stragglers in each cluster by dynamically forming the clusters at each iteration; hence, the proposed scheme is called GC with dynamic clustering (GC-DC).Under a time-correlated straggling behavior, GC-DC gains from adapting to the straggling behavior over time such that, at each iteration, GC-DC aims at distributing the stragglers across clusters as uniformly as possible based on the past straggler behavior. For both homogeneous and heterogeneous worker models, we numerically show that GC-DC provides significant improvements in the average per-iteration completion time without an increase in the communication load compared to the original GC scheme.

show abstract

Irregular Product Coded Computation for High-Dimensional Matrix Multiplication

Cited by 12 publications

References 16 publications

A Survey of Coded Distributed Computing

A Survey of Coded Distributed Computing

Straggler-Aware Distributed Learning: Communication–Computation Latency Trade-Off

Gradient Coding with Dynamic Clustering for Straggler-Tolerant Distributed Learning

Contact Info

Product

Resources

About