Leveraging Coding Techniques for Speeding up Distributed Computing

Abstract

Distributed computing frameworks such as MapReduce are often used to process large computational jobs. They operate by partitioning each job into smaller tasks executed on different servers. The servers also need to exchange intermediate values to complete the computation. Experimental evidence suggests that this so-called Shuffle phase can be a significant part of the overall execution time for several classes of jobs. Prior work has demonstrated a natural tradeoff between computation and communication whereby running redundant copies of jobs can reduce the Shuffle traffic load, thereby leading to reduced overall execution times. For a single job, the main drawback of this approach is that it requires the original job to be split into a number of files that grows exponentially in the system parameters. When extended to multiple jobs (with specific function types), these techniques suffer from a limitation of a similar flavor, i.e., they require an exponentially large number of jobs to be executed. In practical scenarios, these requirements can significantly reduce the promised gains of the method. In this work, we show that a class of combinatorial structures called resolvable designs can be used to develop efficient coded distributed computing schemes for both the single and multiple job scenarios considered in prior work. We present both theoretical analysis and exhaustive experimental results (on Amazon EC2 clusters) that demonstrate the performance advantages of our method. For the single and multiple job cases, we obtain speed-ups of 4.69x (and 2.6x over prior work) and 4.31x over the baseline approach, respectively.

Heterogeneous Coded Distributed Computing: Joint Design of File Allocation and Function Assignment

“…The optimal tradeoff between the computation load in the Map phase and the communication load in the Shuffle phase is derived in [3], which finds that increasing computation load of the Map phase by r can reduce communication load of the Shuffle phase by the same factor r. This idea of coded distributed computing has since been extended widely, e.g., [4]- [9]. In particular, [4], [5] propose new coded distributed computing schemes, [6] studies distributed computing with storage constraints at nodes, [7] studies distributed computing under time-varying excess computing resources, and [8], [9] studies the wireless distributed computing systems.…”

“…, φ Q } with |W k | = W k . Note that, unlike [3]- [6], [8]- [12], W k may vary for different k. Similar to [5], [6], [8]- [12], [14], we assume that W j ∩W k = ∅ for j = k so that each function is assigned to exactly one node. Thus, we have k∈[K] W k = Q.…”

Fundamental Limits of Decentralized Data Shuffling

“…A converse bound was proposed in [14] to show that the proposed coded distributed computing scheme is optimal in terms of communication load. This coded distributed computing framework was extended to the cases such as computing only necessary intermediate values [15], [16], reducing file partitions and number of output functions [16], [17], and considering random network topologies [18], stragglers [19], storage cost [20], and heterogeneous computing power, function assignment and storage space [21], [22].…”

CAMR: Coded Aggregated MapReduce

“…The implicit benefit is that a low requirement on the number of jobs decreases the encoding complexity. This is important since, as we have shown in [7], increasing the number of tasks scales the overhead of the encoding complexity and can diminish any gains in the communication load. We expect a similar type of phenomenon in the current setting.…”

“…By virtue of their simplicity, scalability and fault-tolerance, these frameworks are becoming ubiquitous and have gained significant momentum within both industry and academia. They are well suited for several applications including machine learning [4], [5], graph processing [6], data sorting [7] and web logging [1].…”