Computation Scheduling for Distributed Machine Learning with Straggling Workers

Abstract

We study scheduling of computation tasks across n workers in a large scale distributed learning problem with the help of a master. Computation and communication delays are assumed to be random, and redundant computations are assigned to workers in order to tolerate stragglers. We consider sequential computation of tasks assigned to a worker, while the result of each computation is sent to the master right after its completion. Each computation round, which can model an iteration of the stochastic gradient descent (SGD) algorithm, is completed once the master receives k distinct computations, referred to as the computation target. Our goal is to characterize the average completion time as a function of the computation load, which denotes the portion of the dataset available at each worker, and the computation target. We propose two computation scheduling schemes that specify the tasks assigned to each worker, as well as their computation schedule, i.e., the order of execution. Assuming a general statistical model for computation and communication delays, we derive the average completion time of the proposed schemes. We also establish a lower bound on the minimum average completion time by assuming prior knowledge of the random delays. Experimental results carried out on Amazon EC2 cluster show a significant reduction in the average completion time over existing coded and uncoded computing schemes. It is also shown numerically that the gap between the proposed scheme and the lower bound is relatively small, confirming the efficiency of the proposed scheduling design.

Hierarchical coded matrix multiplication

“…Recent literature (including our work) [12], [1], [21], [2], [3], [4], [22], [23], [24] proposes methods to exploit the work completed by stragglers, rather than ignoring it. The underlying idea is to assign each worker a sequence of multiple small subtasks rather than a single large task.…”

“…We extended special cases of hierarchical coding in [3], [4]. More recent works that aim to exploit stragglers including [23], [24], [22] complement the idea presented in [21] and the idea we present in [1]. In [22] each worker is tasked by a specified fraction of coded and uncoded computations.…”

“…In [23] multiple coded subtasks assigned to each worker are generated according to the characteristics of universally decodable matrices. In [24] each worker is tasked with completing a fully-uncoded series of subtasks with respect to a predesigned computation order.…”

Collaborative Machine Learning at the Wireless Edge with Blind Transmitters

“…where η t denotes the learning rate at iteration t, and shares the result with the devices for the computations at the following iterations. Although parallelism reduces the computation load at each device, communication from the devices to the PS becomes the main performance bottleneck [1]- [5], particularly for wireless edge learning due to limited bandwidth and power. Several architectures have been proposed in recent years to employ computational capabilities of edge devices, and train an ML model collaboratively with the help of a remote PS.…”

Heterogeneous Coded Computation across Heterogeneous Workers

“…Since task completion time depends on the slowest worker, a key bottleneck in distributed computing is the straggler effect: experiments on Amazon EC2 instances show that some workers can be 5 times slower than the typical performance [3]. This Straggler effect can be mitigated by adding redundancy to the distributed computing system via coding [2]- [8], or by scheduling computation tasks [9]- [11]. Maximum distance separable (MDS) codes are widely applied for matrix multiplications [2]- [7], which can reduce the task completion time by O(log N ), where N is the number of workers [2].…”