A Fundamental Tradeoff Between Computation and Communication in Distributed Computing

Li, Songze; Maddah-Ali, Mohammad Ali; Yu, Qian; Avestimehr, A. Salman

doi:10.1109/tit.2017.2756959

Cited by 377 publications

(103 citation statements)

References 40 publications

Supporting

Mentioning

101

Contrasting

Order By: Relevance

“…Before we present the proof of the theorem, we briefly compare our main result with similar results shown in [67], [87]. Our coded shuffling algorithm is related to the coded caching problem [67], since one can design the right cache update rule to reduce the communication rate for an unknown demand or permutation of the data rows.…”

Section: Resultsmentioning

confidence: 69%

“…Thus, the right cache update rule is required to guarantee the opportunity of coded transmission at every iteration. Furthermore, the coded shuffling problem has some connections to coded MapReduce [87] as both algorithms mitigate the communication bottlenecks in distributed computation and machine learning. However, coded shuffling enables coded transmission of raw data by leveraging the extra memory space available at each node, while coded MapReduce enables coded transmission of processed data in the shuffling phase of the MapReduce algorithm by cleverly introducing redundancy in the computation of the mappers.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Speeding up distributed machine learning using codes

Lee

Lam

Pedarsani

et al. 2016

2016 IEEE International Symposium on Information Theory (ISIT)

317

813

View full text Add to dashboard Cite

Codes are widely used in many engineering applications to offer robustness against noise. In large-scale systems there are several types of noise that can affect the performance of distributed machine learning algorithms -straggler nodes, system failures, or communication bottlenecks -but there has been little interaction cutting across codes, machine learning, and distributed systems. In this work, we provide theoretical insights on how coded solutions can achieve significant gains compared to uncoded ones. We focus on two of the most basic building blocks of distributed learning algorithms: matrix multiplication and data shuffling. For matrix multiplication, we use codes to alleviate the effect of stragglers, and show that if the number of homogeneous workers is n, and the runtime of each subtask has an exponential tail, coded computation can speed up distributed matrix multiplication by a factor of log n. For data shuffling, we use codes to reduce communication bottlenecks, exploiting the excess in storage. We show that when a constant fraction α of the data matrix can be cached at each worker, and n is the number of workers, coded shuffling reduces the communication cost by a factor of (α + 1 n )γ(n) compared to uncoded shuffling, where γ(n) is the ratio of the cost of unicasting n messages to n users to multicasting a common message (of the same size) to n users. For instance, γ(n) n if multicasting a message to n users is as cheap as unicasting a message to one user. We also provide experiment results, corroborating our theoretical gains of the coded algorithms.

show abstract

Section: Resultsmentioning

confidence: 69%

Section: Resultsmentioning

confidence: 99%

Speeding up distributed machine learning using codes

Lee

Lam

Pedarsani

et al. 2016

2016 IEEE International Symposium on Information Theory (ISIT)

317

813

View full text Add to dashboard Cite

show abstract

“…Let t * be the solution to the alternative formulation P alt in (7)(8) and τ * be the solution to (16). Then,…”

Section: B Solving the Alternate Formulationmentioning

confidence: 99%

Coded computation over heterogeneous clusters

Reisizadeh

Prakash

Pedarsani

et al. 2017

2017 IEEE International Symposium on Information Theory (ISIT)

108

156

View full text Add to dashboard Cite

In large-scale distributed computing clusters, such as Amazon EC2, there are several types of "system noise" that can result in major degradation of performance: system failures, bottlenecks due to limited communication bandwidth, latency due to straggler nodes, etc. On the other hand, these systems enjoy abundance of redundancy -a vast number of computing nodes and large storage capacity. There have been recent results that demonstrate the impact of coding for efficient utilization of computation and storage redundancy to alleviate the effect of stragglers and communication bottlenecks in homogeneous clusters. In this paper, we focus on general heterogeneous distributed computing clusters consisting of a variety of computing machines with different capabilities. We propose a coding framework for speeding up distributed computing in heterogeneous clusters by trading redundancy for reducing the latency of computation. In particular, we propose Heterogeneous Coded Matrix Multiplication (HCMM) algorithm for performing distributed matrix multiplication over heterogeneous clusters that is provably asymptotically optimal for a broad class of processing time distributions. Moreover, we show that HCMM is unboundedly faster than uncoded schemes that partition the total work load among the workers. To demonstrate how the proposed HCMM scheme can be applied in practice, we provide numerical results demonstrating significant speedups of up to 90% and 35% for HCMM in comparison to the "uncoded" and "coded homogeneous" schemes, respectively. Furthermore, we carry out real experiments over Amazon EC2 clusters that corroborate our numerical studies, where HCMM is found to be up to 17% faster than the uncoded scheme. Additionally, our observation is that machines rarely become stragglers and when they do, they continue to exhibit slower performance for sometime. In our worst case experiments with artificial stragglers, HCMM provides speedups of up to 12× over the uncoded scheme. Furthermore, we provide a generalization of the problem of optimal load allocation for heterogeneous clusters to scenarios with budget constraints and develop a heuristic algorithm for efficient load allocation. In the end, we discuss about the decoding complexity and describe how LDPC codes can be combined with HCMM in order to control the complexity of decoding as the problem size increases.

show abstract

“…The caching problem has also been extended in various directions, including decentralized caching [17], online caching [18], caching with nonuniform demands [19]- [22], hierarchical caching [23]- [25], device-to-device caching [26], cache-aided interference channels [27]- [30], caching on file selection networks [31]- [33], caching on broadcast channels [34]- [37], and caching for channels with delayed feedback with channel state information [38], [39]. The same idea is also useful in the context of distributed computing, in order to take advantage of extra computation to reduce the communication load [40]- [44].…”

Section: Introductionmentioning

confidence: 99%

The Exact Rate-Memory Tradeoff for Caching With Uncoded Prefetching

Maddah-Ali

Avestimehr

2018

IEEE Trans. Inform. Theory

Self Cite

305

278

View full text Add to dashboard Cite

Abstract-We consider a basic cache network, in which a single server is connected to multiple users via a shared bottleneck link. The server has a database of files (content). Each user has an isolated memory that can be used to cache content in a prefetching phase. In a following delivery phase, each user requests a file from the database, and the server needs to deliver users' demands as efficiently as possible by taking into account their cache contents. We focus on an important and commonly used class of prefetching schemes, where the caches are filled with uncoded data. We provide the exact characterization of the rate-memory tradeoff for this problem, by deriving both the minimum average rate (for a uniform file popularity) and the minimum peak rate required on the bottleneck link for a given cache size available at each user. In particular, we propose a novel caching scheme, which strictly improves the state of the art by exploiting commonality among user demands. We then demonstrate the exact optimality of our proposed scheme through a matching converse, by dividing the set of all demands into types, and showing that the placement phase in the proposed caching scheme is universally optimal for all types. Using these techniques, we also fully characterize the rate-memory tradeoff for a decentralized setting, in which users fill out their cache content without any coordination.

show abstract

A Fundamental Tradeoff Between Computation and Communication in Distributed Computing

Cited by 377 publications

References 40 publications

Speeding up distributed machine learning using codes

Speeding up distributed machine learning using codes

Coded computation over heterogeneous clusters

The Exact Rate-Memory Tradeoff for Caching With Uncoded Prefetching

Contact Info

Product

Resources

About