A classification of hadoop job schedulers based on performance optimization approaches

Ghazali, Rana; Adabi, Sahar; Down, Douglas G.; Movaghar, Ali

doi:10.1007/s10586-021-03339-8

Cited by 7 publications

(2 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ghazali et al [23] presented a novel classification system for job schedulers, categorizing them into three distinct groups: job schedulers for mitigating stragglers, job schedulers for enhancing data locality, and job schedulers for optimizing resource utilization. For each job scheduler within these groups, they provided a detailed explanation of their performance-enhancing approach and conducted evaluations to identify their strengths and weaknesses.…”

Section: Regular Surveysmentioning

confidence: 99%

MapReduce scheduling algorithms in Hadoop: a systematic study

Hedayati,

Maleki,

Olsson

et al. 2023

J Cloud Comp

View full text Add to dashboard Cite

Hadoop is a framework for storing and processing huge volumes of data on clusters. It uses Hadoop Distributed File System (HDFS) for storing data and uses MapReduce to process that data. MapReduce is a parallel computing framework for processing large amounts of data on clusters. Scheduling is one of the most critical aspects of MapReduce. Scheduling in MapReduce is critical because it can have a significant impact on the performance and efficiency of the overall system. The goal of scheduling is to improve performance, minimize response times, and utilize resources efficiently. A systematic study of the existing scheduling algorithms is provided in this paper. Also, we provide a new classification of such schedulers and a review of each category. In addition, scheduling algorithms have been examined in terms of their main ideas, main objectives, advantages, and disadvantages.

show abstract

Section: Regular Surveysmentioning

confidence: 99%

MapReduce scheduling algorithms in Hadoop: a systematic study

Hedayati,

Maleki,

Olsson

et al. 2023

J Cloud Comp

View full text Add to dashboard Cite

show abstract

“…Also, there are various job schedulers that consider data locality. These job schedulers can be classified as to whether they consider data locality at the Map task level, data locality at the Reduce task level, or data locality at the job level (both Map and Reduce tasks) [8]. For instance, Hybrid scheduling MapReduce priority (HybSMRP) [9] was presented by Ghandomi et al in 2019 and is a hybrid scheduler that combines dynamic job priority and data localization.…”

Section: Related Workmentioning

confidence: 99%

CLQLMRS: improving cache locality in MapReduce job scheduling using Q-learning

et al. 2022

Self Cite

View full text Add to dashboard Cite

Scheduling of MapReduce jobs is an integral part of Hadoop and effective job scheduling has a direct impact on Hadoop performance. Data locality is one of the most important factors to be considered in order to improve efficiency, as it affects data transmission through the system. A number of researchers have suggested approaches for improving data locality, but few have considered cache locality. In this paper, we present a state-of-the-art job scheduler, CLQLMRS (Cache Locality with Q-Learning in MapReduce Scheduler) for improving both data locality and cache locality using reinforcement learning. The proposed algorithm is evaluated by various experiments in a heterogeneous environment. Experimental results show significantly decreased execution time compared with FIFO, Delay, and the Adaptive Cache Local scheduler.

show abstract

An experimental and comparative benchmark study examining resource utilization in managed Hadoop context

Özdil

Ayvaz

2022

Cluster Comput

View full text Add to dashboard Cite

A classification of hadoop job schedulers based on performance optimization approaches

Cited by 7 publications

References 34 publications

MapReduce scheduling algorithms in Hadoop: a systematic study

MapReduce scheduling algorithms in Hadoop: a systematic study

CLQLMRS: improving cache locality in MapReduce job scheduling using Q-learning

An experimental and comparative benchmark study examining resource utilization in managed Hadoop context

Contact Info

Product

Resources

About