Hierarchical Coding for Distributed Computing

Abstract

Coding for distributed computing supports lowlatency computation by relieving the burden of straggling workers. While most existing works assume a simple master-worker model, we consider a hierarchical computational structure consisting of groups of workers, motivated by the need to reflect the architectures of real-world distributed computing systems. In this work, we propose a hierarchical coding scheme for this model, as well as analyze its decoding cost and expected computation time. Specifically, we first provide upper and lower bounds on the expected computing time of the proposed scheme. We also show that our scheme enables efficient parallel decoding, thus reducing decoding costs by orders of magnitude over non-hierarchical schemes. When considering both decoding cost and computing time, the proposed hierarchical coding is shown to outperform existing schemes in many practical scenarios.

Coded Matrix Multiplication on a Group-Based Model

“…However, the decoding process requires the computational complexity of O(k 3 ). Moreover, the coding schemes suggested in [4], [6], [14] encode the tasks along multiple dimensions, which can effectively reduce the decoding complexities by the virtue of parallel decoding or a peeling decoding scheme. However, these codes lose the MDS property and thereby cannot achieve the optimal computing time.…”

“…We adopt the exponential distribution model for the completion time of a worker, which is defined as the time taken for both the computation and the transmission of the computed result to the master. This model has also been assumed in other papers on coded computation [4], [14]. Unlike these papers, however, a worker in group i has the distribution parameter of µ i , where µ i varies among different groups.…”

“…Afterwards, it is shown that coded computation can effectively improve the performance of computing system with regards to: matrix-matrix multiplication [4]- [6], distributed gradient descent [7], [8], convolution [9], Fourier transform [10], and matrix sparsification [11], [12]. Moreover, regarding the matrix multiplication, new models reflecting the practical environment of computing systems such as the tree structure and heterogeneity are suggested and analyzed [13], [14]. In recent years, distributed cloud computing services such as Amazon EC2 enable customers to deal with large-scale computation [15].…”

“…Moreover, in the real world, the workers' latency statistics are heterogeneous due to a mixed use of hardwares with varying performances or the dynamics of multiple user requests over shared resources [19]. So far, the homogeneous grouped structure has been considered in [14], and the heterogeneous workers without grouped feature has been studied in [13]. However, system solutions which reflect both of the two practical conditions−grouped structure and heterogeneity (in terms of number of workers in each group as well as the bandwidth of the communication links associated with the groups)−are yet to be established.…”

Distributed Gradient Descent with Coded Partial Gradient Computations

“…Coded distributed computation has been introduced to tolerate straggling workers by introducing redundant computations [1][2][3][4][5][6][7][8][9]. Maximum distance separable (MDS) codes are used in [1], where matrix W ∈ R L×L is divided into M disjoint submatrices, W 1 , .…”