Hadoop MapReduce scheduling paradigms

Johannessen, Roger; Yazidi, Anis; Feng, Bin

doi:10.1109/icccbda.2017.7951906

Cited by 6 publications

(2 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They failed to account for the dynamic scheduling [7] associated with operating systems where schedule changes are usually possible only in real-time. So, the underlying scheduling algorithm [8] determines how well the whole system can compute.…”

Section: Reviewmentioning

confidence: 99%

Performance Improvement through Novel Adaptive Node and Container Aware Scheduler with Resource Availability Control in Hadoop YARN

Manjaly,

Subbulakshmi

2023

CSSE

View full text Add to dashboard Cite

The default scheduler of Apache Hadoop demonstrates operational inefficiencies when connecting external sources and processing transformation jobs. This paper has proposed a novel scheduler for enhancement of the performance of the Hadoop Yet Another Resource Negotiator (YARN) scheduler, called the Adaptive Node and Container Aware Scheduler (ANACRAC), that aligns cluster resources to the demands of the applications in the real world. The approach performs to leverage the user-provided configurations as a unique design to apportion nodes, or containers within the nodes, to application thresholds. Additionally, it provides the flexibility to the applications for selecting and choosing which node's resources they want to manage and adds limits to prevent threshold breaches by adding additional jobs as needed. Node or container awareness can be utilized individually or in combination to increase efficiency. On top of this, the resource availability within the node and containers can also be investigated. This paper also focuses on the elasticity of the containers and self-adaptiveness depending on the job type. The results proved that 15%-20% performance improvement was achieved compared with the node and container awareness feature of the ANACRAC. It has been validated that this ANACRAC scheduler demonstrates a 70%-90% performance improvement compared with the default Fair scheduler. Experimental results also demonstrated the success of the enhancement and a performance improvement in the range of 60% to 200% when applications were connected with external interfaces and high workloads.

show abstract

Section: Reviewmentioning

confidence: 99%

Performance Improvement through Novel Adaptive Node and Container Aware Scheduler with Resource Availability Control in Hadoop YARN

Manjaly,

Subbulakshmi

2023

CSSE

View full text Add to dashboard Cite

show abstract

“…By simply disregarding new work that cannot be handled within their timeframe, this algorithm guarantees that jobs with an appropriate deadline are scheduled for completion. This is obtained with the aid of estimating the due date and comparing it to the time it takes to complete the task [45]. In case the schedulability assessment fails, the user will be prompted to set a new due date, and if that is successful, the task will be rescheduled.…”

Section: Deadline Constraint Schedulermentioning

confidence: 99%

An Overview of Hadoop Job Scheduling Algorithms for Big Data

Zameel

Zengi̇n

2022

Mugla Journal of Science and Technology

View full text Add to dashboard Cite

Rapid advancements in Big data systems have occurred over the last several decades. The significant element for attaining high performance is "Job Scheduling" in Big data systems which requires more utmost attention to resolve some challenges of scheduling. To obtain higher performance when processing the big data, proper scheduling is required. Apache Hadoop is most commonly used to manage immense data volumes in an efficient way and also proficient in handling the issues associated with job scheduling. To improve performance of big data systems, we significantly analyzed various Hadoop job scheduling algorithms. To get an overall idea about the scheduling algorithm, this paper presents a rigorous background. This paper made an overview on the fundamental architecture of Hadoop Big data framework, job scheduling and its issues, then reviewed and compared the most important and fundamental Hadoop job scheduling algorithms. In addition, this paper includes a review of other improved algorithms. The primary objective is to present an overview of various scheduling algorithms to improve performance when analyzing big data. This study will also provide appropriate direction in terms of job scheduling algorithm to the researcher according to which characteristics are most significant

show abstract

Analysis of Scheduling Algorithms in Hadoop

Murali

Brindha

2018

Soft Computing Systems

View full text Add to dashboard Cite

Hadoop MapReduce scheduling paradigms

Cited by 6 publications

References 13 publications

Performance Improvement through Novel Adaptive Node and Container Aware Scheduler with Resource Availability Control in Hadoop YARN

Performance Improvement through Novel Adaptive Node and Container Aware Scheduler with Resource Availability Control in Hadoop YARN

An Overview of Hadoop Job Scheduling Algorithms for Big Data

Analysis of Scheduling Algorithms in Hadoop

Contact Info

Product

Resources

About