Coded Elastic Computing

Abstract

Cloud providers have recently introduced new offerings whereby spare computing resources are accessible at discounts compared to on-demand computing. Exploiting such opportunity is challenging inasmuch as such resources are accessed with low-priority and therefore can elastically leave (through preemption) and join the computation at any time. In this paper, we design a new technique called coded elastic computing enabling distributed computations over elastic resources. The proposed technique allows machines to leave the computation without sacrificing the algorithm-level performance, and, at the same time, adaptively reduce the workload at existing machines when new ones join the computation. Leveraging coded redundancy, our approach is able to achieve similar computational cost as the original (uncoded) method when all machines are present; the cost gracefully increases when machines are preempted and reduces when machines join. The performance of the proposed technique is evaluated on matrix-vector multiplication and linear regression tasks. In experimental validations, it can achieve exactly the same numerical result as the noiseless computation, while reducing the computation time by 46% when compared to non-adaptive coding schemes.

A Practical Algorithm Design and Evaluation for Heterogeneous Elastic Computing with Stragglers

“…Coded Storage Elastic Computing (CSEC) was introduced by Yang et al [1] as an effective method to perform distributed computing on elastic cloud systems where machines have limited storage capacity. Here, elasticity means that machines can join and leave the network periodically as machines may be reserved or freed from other higher priority jobs.…”

“…The overall computation is split into many steps, where in each step we know which machines are available, even though their availability may change over computation steps. The original CSEC design [1] studied homogeneous CSEC systems where machines have the same computation speed. It proposed a cyclic homogeneous computation assignment design to minimize the computation load at each machine such that no computation redundancy is present in the network.…”

“…Based on our measurements of the computation speeds on Amazon EC2 instances, we found that these virtual instances often have different computation speeds even if they share the same configuration. Hence, in [15], for heterogeneous elastic computing systems where machines have different computation speeds, we reformulated the CSEC problem as a combinatorial optimization problem with the objective of minimizing the overall computation time, instead of per machine computation load as in [1]. This combinatorial optimization problem was solved using a novel approach, which decomposed this problem into two sub-optimization problems consisting of a convex optimization problem and a combinatorial "filling" problem, where an optimal solution was found.…”

“…Adding straggler tolerance in CSEC is an interesting problem that has not been studied in the literature. Given the MDS coded storage placement used in [1], Kiani et al in [17] proposed hierarchical computation assignment algorithms for homogeneous CSEC systems. However, the performance evaluation in [17] were done using computer simulations, where stragglers are modeled to take place with certain probabilities, instead of using practical cloud computing systems such as Amazon EC2 or Microsoft Azure.…”

Straggler-Aware Distributed Learning: Communication–Computation Latency Trade-Off

“…A wealth of straggler avoidance techniques have been proposed in recent years for DGD as well as other distributed computation tasks [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 ]. The common design notion behind all these schemes is the assignment of redundant computations/tasks to workers, such that faster workers can compensate for the stragglers.…”