On the Fundamental Limits of Coded Data Shuffling for Distributed Machine Learning

Elmahdy, Adel M.; Mohajer, Soheil

doi:10.1109/tit.2020.2964547

Cited by 14 publications

(15 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Elmahdy and Mohajer considered the data shuffling problem that a master node communicates a set of files to a set of worker nodes through a shared link. They proposed a deterministic and systematic coded shuffling scheme to find out the exact rate of cache files . These researched results support our work to reduce shuffling overloads, include computing and communication overhead, through an execution instructions redesigning methods.…”

Section: Related Worksupporting

confidence: 71%

“…They proposed a deterministic and systematic coded shuffling scheme to find out the exact rate of cache files. 31,32 These researched results support our work to reduce shuffling overloads, include computing and communication overhead, through an execution instructions redesigning methods. We analyze the programming language and algorithm structure to practically design a set of replacement instructions and cache the exact set of data for iterative applications from implementation perspective.…”

Section: Related Work Comparisonssupporting

confidence: 68%

See 1 more Smart Citation

Performance enhancement for iterative data computing with in‐memory concurrent processing

Wen

Chen

Chiu

et al. 2019

Concurrency and Computation

View full text Add to dashboard Cite

Summary The big data era has resulted in the development of several data analysis tools. Spark is a type of in‐memory processing fitted iteration and interactive data mining tool. This tool possesses higher data‐processing performance than MapReduce, which is an offline storage mechanism. However, some disadvantages of in‐memory processing, such as massive in‐memory data requirements, cause cross‐node data transfer that result in a long computation time. The performance of the process can be improved if the in‐memory process is executed with fewer shuffle instructions. Therefore, this study aims to enhance the performance of iterative application through instruction replacement. Three empirical research cases with diverse datasets and iterations are used to modify the program. We adopt a strategy of downloading a small resilient distributed dataset and replacing the shuffle‐included instructions to shorten the processing time with an automated code replacement by using exhaustively code matching. The experimental results reveal an improvement of up to 39% in the execution time compared with the existing in‐memory processing programs with various dataset sizes.

show abstract

Section: Related Worksupporting

confidence: 71%

Section: Related Work Comparisonssupporting

confidence: 68%

Performance enhancement for iterative data computing with in‐memory concurrent processing

Wen

Chen

Chiu

et al. 2019

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…The above data exchange problem models a number of cache-enabled multi-receiver communication problems studied recently in the coding theory community, including Coded Caching [1], Coded Distributed Computing [2,3], Coded Data Shuffling [4][5][6], and Coded Data Rebalancing [7]. In [8], a special case of our general problem here was considered in the name of cooperative data exchange, where the goal was to reach a state in which all nodes have all the data in the system.…”

Section: Introduction and Main Resultsmentioning

confidence: 99%

“…We have considered the decentralized version of the coded data shuffling problem in this subsection. The centralized version of the data shuffling problem was introduced in [4] and its information theoretic limits were studied elaborately in [6]. Our data exchange bound, when applied to the setting in [6], results in a looser converse result than that in [6].…”

Section: Proof Of the Decentralized Data Shuffling Conversementioning

confidence: 99%

“…The centralized version of the data shuffling problem was introduced in [4] and its information theoretic limits were studied elaborately in [6]. Our data exchange bound, when applied to the setting in [6], results in a looser converse result than that in [6]. The reasons for this is explored in Section 5 using the connection between our data exchange bound and the bound for index coding known in literature.…”

Section: Proof Of the Decentralized Data Shuffling Conversementioning

confidence: 99%

See 1 more Smart Citation

An Umbrella Converse for Data Exchange: Applied to Caching, Computing, and Shuffling

Krishnan

Natarajan

Lalitha

2021

Entropy

View full text Add to dashboard Cite

The problem of data exchange between multiple nodes with storage and communication capabilities models several current multi-user communication problems like Coded Caching, Data Shuffling, Coded Computing, etc. The goal in such problems is to design communication schemes which accomplish the desired data exchange between the nodes with the optimal (minimum) amount of communication load. In this work, we present a converse to such a general data exchange problem. The expression of the converse depends only on the number of bits to be moved between different subsets of nodes, and does not assume anything further specific about the parameters in the problem. Specific problem formulations, such as those in Coded Caching, Coded Data Shuffling, and Coded Distributed Computing, can be seen as instances of this generic data exchange problem. Applying our generic converse, we can efficiently recover known important converses in these formulations. Further, for a generic coded caching problem with heterogeneous cache sizes at the clients with or without a central server, we obtain a new general converse, which subsumes some existing results. Finally we relate a “centralized” version of our bound to the known generalized independence number bound in index coding and discuss our bound’s tightness in this context.

show abstract

Why Globally Re-shuffle? Revisiting Data Shuffling in Large Scale Deep Learning

Nguyen

Domke

Drozd

et al. 2022

2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

View full text Add to dashboard Cite

Stochastic gradient descent (SGD) is the most prevalent algorithm for training Deep Neural Networks (DNN). SGD iterates the input data set in each training epoch processing data samples in a random access fashion. Because this puts enormous pressure on the I/O subsystem, the most common approach to distributed SGD in HPC environments is to replicate the entire dataset to node local SSDs. However, due to rapidly growing data set sizes this approach has become increasingly infeasible. Surprisingly, the questions of why and to what extent random access is required have not received a lot of attention in the literature from an empirical standpoint.In this paper, we revisit data shuffling in DL workloads to investigate the viability of partitioning the dataset among workers and performing only a partial distributed exchange of samples in each training epoch. Through extensive experiments on up to 2,048 GPUs of ABCI and 4,096 compute nodes of Fugaku, we demonstrate that in practice validation accuracy of global shuffling can be maintained when carefully tuning the partial distributed exchange. We provide a solution implemented in PyTorch that enables users to control the proposed data exchange scheme.

show abstract

On the Fundamental Limits of Coded Data Shuffling for Distributed Machine Learning

Cited by 14 publications

References 23 publications

Performance enhancement for iterative data computing with in‐memory concurrent processing

Performance enhancement for iterative data computing with in‐memory concurrent processing

An Umbrella Converse for Data Exchange: Applied to Caching, Computing, and Shuffling

Why Globally Re-shuffle? Revisiting Data Shuffling in Large Scale Deep Learning

Contact Info

Product

Resources

About