A Data Distribution Aware Task Scheduling Strategy for MapReduce System

Guo, Leitao; Sun, Hongwei; Luo, Zhiguo

doi:10.1007/978-3-642-10665-1_74

Cited by 7 publications

(3 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Guo et al [229] proposed a data distribution aware task scheduling methodology by overcoming the uneven data distribution strategy of default scheduling techniques. The network overhead is reduced and high system efficiency is reported by mapping tasks to nodes with high probability of data availability and task's scheduling priority.…”

Section: Adoptive Schedulingmentioning

confidence: 99%

Data Locality in High Performance Computing, Big Data, and Converged Systems: An Analysis of the Cutting Edge and A Future System Architecture

Usman¹,

Mehmood²,

Katib³

et al. 2022

Preprint

View full text Add to dashboard Cite

Big data has revolutionised science and technology leading to the transformation of our societies. High Performance Computing (HPC) provides the necessary computational power for big data analysis using artificial intelligence and methods. Traditionally HPC and big data had focused on different problem domains and had grown into two different ecosystems. Efforts have been underway for the last few years on bringing the best of both paradigms into HPC and big converged architectures. Designing HPC and big data converged systems is a hard task requiring careful placement of data, analytics, and other computational tasks such that the desired performance is achieved with the least amount of resources. Energy efficiency has become the biggest hurdle in the realisation of HPC, big data, and converged systems capable of delivering exascale and beyond performance. Data locality is a key parameter of HPDA system design as moving even a byte costs heavily both in time and energy with an increase in the size of the system. Performance in terms of time and energy are the most important factors for users, particularly energy, due to it being the major hurdle in high performance system design and the increasing focus on green energy systems due to environmental sustainability. Data locality is a broad term that encapsulates different aspects including bringing computations to data, minimizing data movement by efficient exploitation of cache hierarchies, reducing intra- and inter-node communications, locality-aware process and thread mapping, and in-situ and in-transit data analysis. This paper provides an extensive review of the cutting-edge on data locality in HPC, big data, and converged systems. We review the literature on data locality in HPC, big data, and converged environments and discuss challenges, opportunities, and future directions. Subsequently, using the knowledge gained from this extensive review, we propose a system architecture for future HPC and big data converged systems. To the best of our knowledge, there is no such review on data locality in converged HPC and big data systems.

show abstract

Section: Adoptive Schedulingmentioning

confidence: 99%

Data Locality in High Performance Computing, Big Data, and Converged Systems: An Analysis of the Cutting Edge and A Future System Architecture

Usman¹,

Mehmood²,

Katib³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Guo et al [229] proposed a data-distribution-aware task scheduling methodology by overcoming the uneven data-distribution strategy of default scheduling techniques. The network overhead is reduced and high system efficiency is reported by mapping tasks to nodes with a high probability of data availability and the task's scheduling priority.…”

Section: Adaptive Schedulingmentioning

confidence: 99%

Data Locality in High Performance Computing, Big Data, and Converged Systems: An Analysis of the Cutting Edge and a Future System Architecture

Usman¹,

Mehmood

Katib

et al. 2022

Electronics

View full text Add to dashboard Cite

Big data has revolutionized science and technology leading to the transformation of our societies. High-performance computing (HPC) provides the necessary computational power for big data analysis using artificial intelligence and methods. Traditionally, HPC and big data had focused on different problem domains and had grown into two different ecosystems. Efforts have been underway for the last few years on bringing the best of both paradigms into HPC and big converged architectures. Designing HPC and big data converged systems is a hard task requiring careful placement of data, analytics, and other computational tasks such that the desired performance is achieved with the least amount of resources. Energy efficiency has become the biggest hurdle in the realization of HPC, big data, and converged systems capable of delivering exascale and beyond performance. Data locality is a key parameter of HPDA system design as moving even a byte costs heavily both in time and energy with an increase in the size of the system. Performance in terms of time and energy are the most important factors for users, particularly energy, due to it being the major hurdle in high-performance system design and the increasing focus on green energy systems due to environmental sustainability. Data locality is a broad term that encapsulates different aspects including bringing computations to data, minimizing data movement by efficient exploitation of cache hierarchies, reducing intra- and inter-node communications, locality-aware process and thread mapping, and in situ and transit data analysis. This paper provides an extensive review of cutting-edge research on data locality in HPC, big data, and converged systems. We review the literature on data locality in HPC, big data, and converged environments and discuss challenges, opportunities, and future directions. Subsequently, using the knowledge gained from this extensive review, we propose a system architecture for future HPC and big data converged systems. To the best of our knowledge, there is no such review on data locality in converged HPC and big data systems.

show abstract

“…Otherwise, the method schedules the selected task to the node. In [13], the authors offered a data distribution aware task scheduling strategy for MapReduce system. Their strategy has two main phases: In the initialization phase, statistics number of copies of data processed by each map task.…”

Section: Distribution Aware Schedulingmentioning

confidence: 99%

A Comprehensive View of MapReduce Aware Scheduling Algorithms in Cloud Environments

Yazdanpanah¹,

Shouraki²,

Abshirini³

2015

IJCA

View full text Add to dashboard Cite

Cloud computing has emerged as a model that harnesses massive capacities of data centers to host services in a costeffective manner. MapReduce has been widely used as a Big Data processing platform, proposed by Google in 2004 and has become a popular parallel computing framework for large-scale data processing since then. It is best suited for embarrassingly parallel and data-intensive tasks. It is designed to read large amount of data stored in a distributed file system such as Google File System (GFS), process the data in parallel, aggregate and store the results back to the distributed file system. Scheduling is one of the most critical aspects of MapReduce. Also three important scheduling issues in MapReduce such as locality, synchronization and fairness exist. This paper tries to illustrate and analyze the overview of thirteen different aware scheduling algorithms with different techniques and approaches for MapReduce in Hadoop and their scheduling issues and problems. At the end, Advantages and disadvantages of these algorithms are identified.

show abstract

A Data Distribution Aware Task Scheduling Strategy for MapReduce System

Cited by 7 publications

References 1 publication

Data Locality in High Performance Computing, Big Data, and Converged Systems: An Analysis of the Cutting Edge and A Future System Architecture

Data Locality in High Performance Computing, Big Data, and Converged Systems: An Analysis of the Cutting Edge and A Future System Architecture

Data Locality in High Performance Computing, Big Data, and Converged Systems: An Analysis of the Cutting Edge and a Future System Architecture

A Comprehensive View of MapReduce Aware Scheduling Algorithms in Cloud Environments

Contact Info

Product

Resources

About