Job Aware Scheduling Algorithm for MapReduce Framework

Nanduri, Radheshyam; Maheshwari, Nitesh; Reddyraja, A.; Varma, Vasudeva

doi:10.1109/cloudcom.2011.112

Cited by 46 publications

(24 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The performance of MapReduce is primarily dependent on its task scheduler [11], minimizing the overall completion time of a job by appropriately assigning tasks to the available nodes is a common goal of the MapReduce scheduling. In Hadoop cluster, if a task is executing for longer period of time compared to other tasks then this condition will be called as straggler.…”

Section: Mapreduce Scheduling In Heterogeneous Environmentmentioning

confidence: 99%

Performance Improvement of MapReduce Framework in Heterogeneous Context using Reinforcement Learning

Naik

Negi

Sastry

2015

Procedia Computer Science

View full text Add to dashboard Cite

MapReduce is presently established as an important distributed and parallel programming model with wide acclaim for large scale computing. Intelligent scheduling decisions can help in reducing the overall runtime of the jobs. MapReduce performance is currently limited by its default scheduler, which does not adapt well in heterogeneous environments. Heterogeneous environments were considered in Longest Approximate Time to End scheduler. This too has several shortcomings due to the static manner in which it computes progress of tasks. The lack of adequate approach to heterogeneous environments is currently being taken up in recent research. In this paper, we propose a novel MapReduce scheduler in heterogeneous environments based on Reinforcement learning called MapReduce Reinforcement Learning scheduler, which observes the system state of task execution and suggests speculative re-execution of the slower tasks to other available nodes in the cluster for faster execution. The proposed approach adapts to the heterogeneous environment and no prior knowledge of the environmental characteristics are required. It is expected that over a few runs the system would be able to better map the computing requirements to the resources available in a heterogeneous cluster and minimizes the overall job completion time.c 2015 The Authors. Published by Elsevier B.V. Peer-review under responsibility of scientific committee of 2nd International Symposium on Big Data and Cloud Computing (ISBCC'15).

show abstract

Section: Mapreduce Scheduling In Heterogeneous Environmentmentioning

confidence: 99%

Performance Improvement of MapReduce Framework in Heterogeneous Context using Reinforcement Learning

Naik

Negi

Sastry

2015

Procedia Computer Science

View full text Add to dashboard Cite

show abstract

“…These issues have been undervalued by researchers in most of the proposed MapReduce scheduling algorithms [9], which leads to the poor performance of Hadoop [10]. Minimizing the execution time of a job by appropriately assigning tasks to the available heterogeneous nodes in the cluster is a common goal of the MapReduce scheduler [11] and it is likewise a significant research topic because it enhances the performance of MapReduce framework.…”

Section: Introductionmentioning

confidence: 98%

Enhancing the Performance of MapReduce Default Scheduler by Detecting Prolonged TaskTrackers in Heterogeneous Environments

Naik

Negi

Sastry

2015

Advances in Intelligent Systems and Computing

View full text Add to dashboard Cite

MapReduce is now a significant parallel processing model for large-scale data-intensive applications using clusters with commodity hardware. Scheduling of jobs and tasks, and identification of TaskTrackers which are slow in Hadoop clusters are the focus research in the recent years. MapReduce performance is currently limited by its default scheduler, which does not adapt well in heterogeneous environments. In this paper, we propose a scheduling method to identify the TaskTrackers which are running slowly in map and reduce phases of the MapReduce framework in a heterogeneous Hadoop cluster. The proposed method is integrated with the MapReduce default scheduling algorithm. The performance of this method is compared with the unmodified MapReduce default scheduler. We observe that the proposed approach shows improvements in performance to the default scheduler in the heterogeneous environments. Performance improvement was observed as the overall job execution times for different workloads from HiBench benchmark suite were reduced.

show abstract

“…ESAMR records historical information for each node as in case of SAMR and it adopts a k-means clustering algorithm to dynamically tune stage weight parameters and to find slow tasks accurately. ESAMR significantly improves the performance of MapReduce scheduling in terms of estimating task execution time and launching backup tasks [9].…”

Section: Esamr: An Enhanced Self-adaptive Mapreduce Scheduling Algorithmmentioning

confidence: 99%

Survey on MapReduce Scheduling Algorithms

Thomas¹,

Syama²

2014

IJCA

View full text Add to dashboard Cite

MapReduce is a programming model used by Google to process large amount of data in a distributed computing environment. It is usually used to perform distributed computing on clusters of computers. Computational processing of data stored on either a file system or a database usually occurs. MapReduce takes the advantage of locality of data, processing data on or near the storage areas, thereby avoiding unnecessary data transmission. The simplicity of the programming model and the automatic handling of node failures hiding the complexity of fault tolerance make MapReduce to be used for both commercial and scientific applications. As MapReduce clusters have become popular these days, their scheduling is one of the important factor which is to be considered. In order to achieve good performance a MapReduce scheduler must avoid unnecessary data transmission. Hence different scheduling algorithms for MapReduce are necessary to provide good performance. This paper provides an overview of four different scheduling algorithms for MapReduce namely; Scheduling algorithm in Hadoop, Longest Approximate Time to End (LATE) MapReduce scheduling algorithm, Self-Adaptive MapReduce(SAMR) scheduling algorithm and Enhanced Self-Adaptive MapReduce scheduling algorithm(ESAMR). An overview of these techniques is provided through this paper. Advantages and disadvantages of these algorithms are identified.

show abstract

Job Aware Scheduling Algorithm for MapReduce Framework

Cited by 46 publications

References 7 publications

Performance Improvement of MapReduce Framework in Heterogeneous Context using Reinforcement Learning

Performance Improvement of MapReduce Framework in Heterogeneous Context using Reinforcement Learning

Enhancing the Performance of MapReduce Default Scheduler by Detecting Prolonged TaskTrackers in Heterogeneous Environments

Survey on MapReduce Scheduling Algorithms

Contact Info

Product

Resources

About