Fundamental tradeoff between computation and communication in distributed computing

Abstract

Abstract-How can we optimally trade extra computing power to reduce the communication load in distributed computing? We answer this question by characterizing a fundamental tradeoff between computation and communication in distributed computing, i.e., the two are inversely proportional to each other.More specifically, a general distributed computing framework, motivated by commonly used structures like MapReduce, is considered, where the overall computation is decomposed into computing a set of "Map" and "Reduce" functions distributedly across multiple computing nodes. A coded scheme, named "Coded Distributed Computing" (CDC), is proposed to demonstrate that increasing the computation load of the Map functions by a factor of r (i.e., evaluating each function at r carefully chosen nodes) can create novel coding opportunities that reduce the communication load by the same factor.An information-theoretic lower bound on the communication load is also provided, which matches the communication load achieved by the CDC scheme. As a result, the optimal computation-communication tradeoff in distributed computing is exactly characterized.Finally, the coding techniques of CDC is applied to the Hadoop TeraSort benchmark to develop a novel CodedTeraSort algorithm, which is empirically demonstrated to speed up the overall job execution by 1.97× -3.39×, for typical settings of interest.

Energy-Efficient Edge-Facilitated Wireless Collaborative Computing using Map-Reduce

“…Next, inequalities (4), (7), (8), and (15) are either linear or affine and thus define a polyhedron. The only remaining constraint (omitting trivial positivity constraints on all variables) is then constraint (14). For constraint (14) to define a convex set, its right-hand side term must be a concave function.…”

“…The only remaining constraint (omitting trivial positivity constraints on all variables) is then constraint (14). For constraint (14) to define a convex set, its right-hand side term must be a concave function. The function r n (x) is a concave function with respect to x ≥ 0.…”

“…Increasing µ n has a double effect: (i) it decreases the value of l * n because l * n is proportional to √ λ − αµ n , and (ii) it increases the value of p * n because p * n is directly proportional to µ n . Combined, these two effects work together towards satisfying the rate constraint (14). For nodes with very bad channel or very low maximum RF transmit power p max n , µ n could increase so much that λ − αµ n would become negative, meaning that the number of bits to be processed l * n would fall to 0.…”

“…For nodes with very bad channel or very low maximum RF transmit power p max n , µ n could increase so much that λ − αµ n would become negative, meaning that the number of bits to be processed l * n would fall to 0. At this point, there is an iterative interplay between λ and {µ n } N n=1 in which both successively increase to maximize the dual function until both constraints (1) and (14) are satisfied and a maximum has been reached.…”

Performance enhancement for iterative data computing with in‐memory concurrent processing

“…The shuffle operation results in high cost (i.e., blocks processing, network queuing, and RDDs transmitting) between the local and remote file systems when the running program initially joins data from offline storage or updates intermediate data results . Although the cited paper focused on the MapReduce framework, the decentralized operation concept caused a fundamental tradeoff between computation and transmission in decentralized environment is considered in this work. The detail duty cycle is shown in Figure that the shuffle time includes host processing time, network handle time, and data transmission time.…”