Optimal Load Allocation for Coded Distributed Computation in Heterogeneous Clusters

Abstract

Recently, coding has been a useful technique to mitigate the effect of stragglers in distributed computing.However, coding in this context has been mainly explored under the assumption of homogeneous workers, although the real-world computing clusters can be often composed of heterogeneous workers that have different computing capabilities. The uniform load allocation without the awareness of heterogeneity possibly causes a significant loss in latency. In this paper, we suggest the optimal load allocation for coded distributed computing with heterogeneous workers. Specifically, we focus on the scenario that there exist workers having the same computing capability, which can be regarded as a group for analysis. We rely on the lower bound on the expected latency and obtain the optimal load allocation by showing that our proposed load allocation achieves the minimum of the lower bound for a sufficiently large number of workers. From numerical simulations, when assuming the group heterogeneity, our load allocation reduces the expected latency by orders of magnitude over the existing load allocation scheme.

A Survey of Coded Distributed Computing

“…In practical distributed computing systems, some processing nodes have the same computational capabilities, in terms of the same distributions of computation time, and thus they can be grouped together. By exploiting the group structure and heterogeneities among different groups of processing nodes [141], [142], the implementation of a combination of group codes and an optimal load allocation strategy not only approaches the optimal computation time that is achieved by the MDS codes, but also has low decoding complexity. In addition, by varying the number of allocated rows of the matrix to the workers [142], the computation latency can be reduced by orders of magnitude over the MDS codes with fixed computation load allocation [141] as the number of workers increases.…”

“…By exploiting the group structure and heterogeneities among different groups of processing nodes [141], [142], the implementation of a combination of group codes and an optimal load allocation strategy not only approaches the optimal computation time that is achieved by the MDS codes, but also has low decoding complexity. In addition, by varying the number of allocated rows of the matrix to the workers [142], the computation latency can be reduced by orders of magnitude over the MDS codes with fixed computation load allocation [141] as the number of workers increases. The load allocation strategy proposed in [142] focuses mainly on the design of an optimal MDS code.…”

“…In addition, by varying the number of allocated rows of the matrix to the workers [142], the computation latency can be reduced by orders of magnitude over the MDS codes with fixed computation load allocation [141] as the number of workers increases. The load allocation strategy proposed in [142] focuses mainly on the design of an optimal MDS code. Other efficient coding algorithms such as LT codes [74] can also be explored in future studies.…”

Robust incentive mechanisms for efficient and secure coded edge intelligence

“…In practical distributed computing systems, some computing nodes have the same computational capabilities, in terms of the same distributions of computation time, and thus they can be grouped together. By exploiting the group structure and heterogeneities among different groups of computing nodes [108], [109], the implementation of a combination of group codes and an optimal load allocation strategy not only approaches the optimal computation time that is achieved by the MDS codes, but also has low decoding complexity. In addition, by varying the number of allocated rows of the matrix to the workers [109], the computation latency can be reduced by orders of magnitude over the MDS codes with fixed computation load allocation [108] as the number of workers increases.…”

“…By exploiting the group structure and heterogeneities among different groups of computing nodes [108], [109], the implementation of a combination of group codes and an optimal load allocation strategy not only approaches the optimal computation time that is achieved by the MDS codes, but also has low decoding complexity. In addition, by varying the number of allocated rows of the matrix to the workers [109], the computation latency can be reduced by orders of magnitude over the MDS codes with fixed computation load allocation [108] as the number of workers increases. The load allocation strategy proposed in [109] focuses mainly on the design of an optimal MDS code.…”