Hierarchical Coded Matrix Multiplication

Kiani, Shahrzad; Ferdinand, Nuwan S.; Draper, Stark C.

doi:10.1109/tit.2020.3036763

Cited by 30 publications

(15 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that with identical redundancy, the computing capabilities of partial stragglers cannot be effectively utilized since incomplete computation results from stragglers are wasted. To avoid wasting incomplete computation results, in [12]- [14], [16], [20], [21], diverse redundancies are introduced in coded computation schemes, and incomplete computation results such as coded submatrix-submatrix products corresponding to some data blocks [4], [5], [9]- [11], [15], [22], [23] and coded local partial derivatives corresponding to some coordinates [6]- [8], [17]- [19] are utilized. Note that in [12], [16], [20], [21], coding parameters determining the amount of redundancies are artificially fixed, which may limit the performances of coded computation schemes.…”

Section: Introductionmentioning

confidence: 99%

Optimization-based Block Coordinate Gradient Coding for Mitigating Partial Stragglers in Distributed Learning

Wang¹,

Cui²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Gradient coding schemes effectively mitigate full stragglers in distributed learning by introducing identical redundancy in coded local partial derivatives corresponding to all model parameters. However, they are no longer effective for partial stragglers as they cannot utilize incomplete computation results from partial stragglers. This paper aims to design a new gradient coding scheme for mitigating partial stragglers in distributed learning. Specifically, we consider a distributed system consisting of one master and N workers, characterized by a general partial straggler model and focuses on solving a general large-scale machine learning problem with L model parameters using gradient coding. First, we propose a coordinate gradient coding scheme with L coding parameters representing L possibly different diversities for the L coordinates, which generates most gradient coding schemes. Then, we consider the minimization of the expected overall runtime and the maximization of the completion probability with respect to the L coding parameters for coordinates, which are challenging discrete optimization problems. To reduce computational complexity, we first transform each to an equivalent but much simpler discrete problem with N L variables representing the partition of the L coordinates into N blocks, each with identical redundancy. This indicates an equivalent but more easily implemented block coordinate gradient coding scheme with N coding parameters for blocks. Then, we adopt continuous relaxation to further reduce computational complexity. For the resulting minimization of expected overall runtime, we develop an iterative algorithm of computational complexity O(N 2 ) to obtain an optimal solution and derive two closed-form approximate solutions both with computational complexity O(N ). For the resultant maximization of the completion probability, we develop an iterative algorithm of computational complexity O(N 2 ) to obtain a stationary point and derive a closed-form approximate solution with computational complexity O(N ) at a large threshold. Finally, numerical results show that the proposed solutions significantly outperform existing coded computation schemes and their extensions.

show abstract

Section: Introductionmentioning

confidence: 99%

Optimization-based Block Coordinate Gradient Coding for Mitigating Partial Stragglers in Distributed Learning

Wang¹,

Cui²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Thus, designing schemes that combine few submatrices while continuing to have a good threshold is important. Secondly, most of the previous works (see [13], [14], [15], [16], [17], [18], [19], [20] for some exceptions) treat the stragglers as erasures, and thus discard the computations done by the slower workers. But a slower worker may not be a useless worker and approaches which can efficiently utilize the partial computations done by the slower workers are of interest.…”

Section: Introductionmentioning

confidence: 99%

A Unified Treatment of Partial Stragglers and Sparse Matrices in Coded Matrix Computation

Das¹,

Ramamoorthy²

2021

Preprint

View full text Add to dashboard Cite

The overall execution time of distributed matrix computations is often dominated by slow worker nodes (stragglers) over the clusters. Recently, different coding techniques have been utilized to mitigate the effect of stragglers where worker nodes are assigned the task of processing encoded submatrices of the original matrices. In many machine learning or optimization problems the relevant matrices are often sparse. Several coded computation methods operate with dense linear combinations of the original submatrices; this can significantly increase the worker node computation times and consequently the overall job execution time. Moreover, several existing techniques treat the stragglers as failures (erasures) and discard their computations. In this work, we present a coding approach which operates with limited encoding of the original submatrices and utilizes the partial computations done by the slower workers. Our scheme continues to have the optimal threshold of prior work. Extensive numerical experiments done in AWS (Amazon Web Services) cluster confirm that the proposed approach enhances the speed of the worker computations (and thus the whole process) significantly.

show abstract

“…A significant amount of computation power is wasted. To handle this situation, a setting in which the number of stragglers is not known a priori has been considered in [35]- [47] and schemes that can cope with such a setting have been designed. The underlying idea is to assign a sequence of small tasks to each server instead of assigning a single large task.…”

Section: Introductionmentioning

confidence: 99%

“…References [41]- [46] consider matrix-matrix multiplication, but they can only handle a special partitioning, i.e., A and B are row-wisely and column-wisely split, respectively, or only A is row-wisely split. In [47], the authors propose 3 hierarchical schemes for matrix multiplication to leverage partial stragglers. The main idea is that the task is first divided into several small subtasks, i.e., the multiplication of several pairs of small matrices, and each subtask is coded separately with existing schemes.…”

Section: Introductionmentioning

confidence: 99%

“…Therefore, the number of available servers is flexible and the number of required tasks is adjusted to the number of available servers. Our scheme is different from those that leverage partial stragglers [39], [40], [42], [45]- [47]. In our construction, the computation load (the number of multiplication operations) of each non-straggler server is the same and the computation by stragglers is neglected, while in schemes with partial stragglers, the computation load varies in different servers including stragglers.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Flexible Distributed Matrix Multiplication

Li¹,

Chen²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

The distributed matrix multiplication problem with an unknown number of stragglers is considered, where the goal is to efficiently and flexibly obtain the product of two massive matrices by distributing the computation across N servers. There are up to N − R stragglers but the exact number is not known a priori. Motivated by reducing the computation load of each server, a flexible solution is proposed to fully utilize the computation capability of available servers. The computing task for each server is separated into several subtasks, constructed based on Entangled Polynomial codes by Yu et al. The final results can be obtained from either a larger number of servers with a smaller amount of computation completed per server or a smaller number of servers with a larger amount of computation completed per server. The required finite field size of the proposed solution is less than 2N . Moreover, the optimal design parameters such as the partitioning of the input matrices is discussed.Our constructions can also be generalized to other settings such as batch distributed matrix multiplication and secure distributed matrix multiplication.

show abstract

Hierarchical Coded Matrix Multiplication

Cited by 30 publications

References 24 publications

Optimization-based Block Coordinate Gradient Coding for Mitigating Partial Stragglers in Distributed Learning

Optimization-based Block Coordinate Gradient Coding for Mitigating Partial Stragglers in Distributed Learning

A Unified Treatment of Partial Stragglers and Sparse Matrices in Coded Matrix Computation

Flexible Distributed Matrix Multiplication

Contact Info

Product

Resources

About