ATLAS: An AdapTive faiLure-Aware Scheduler for Hadoop

Abstract

Hadoop has become the de facto standard for processing large data in today's cloud environment. The performance of Hadoop in the cloud has a direct impact on many important applications ranging from web analytic, web indexing, image and document processing to high-performance scientific computing. However, because of the scale, complexity and dynamic nature of the cloud, failures are common and these failures often impact the performance of jobs running in Hadoop. Although Hadoop possesses built-in failure detection and recovery mechanisms, several scheduled jobs still fail because of unforeseen events in the cloud environment. A single task failure can cause the failure of the whole job and unpredictable job running times. In this paper, we propose ATLAS (AdapTive faiLure-Aware Scheduler), a new scheduler for Hadoop that can adapt its scheduling decisions to events occurring in the cloud environment. Using statistical models, ATLAS predicts task failures and adjusts its scheduling decisions on the fly to reduce task failure occurrences. We implement ATLAS in the Hadoop framework of Amazon Elastic MapReduce (EMR) and perform a case study to compare its performance with those of the FIFO, Fair and Capacity schedulers. Results show that ATLAS can reduce the percentage of failed jobs by up to 28% and the percentage of failed tasks by up to 39%, and the total execution time of jobs by 10 minutes on average. ATLAS also reduces CPU and memory usages.

Experimental Analysis in Hadoop MapReduce: A Closer Look at Fault Detection and Recovery Techniques

“…In another study, Memishi et al [ 9 ] also proposed an approach that estimates the completion time of the workload and calculates the progress rate of each task to adjust the timeout value dynamically. Other studies by [ 17 , 18 , 19 , 20 ] provided predictive models based on machine learning and AI algorithms to estimate and set an optimal heartbeat timeout on the fly or to predict the failures before they occur. These approaches reduce the task fault occurrences and improve their overall performance with low latency in fault detection.…”

Task Scheduling in Big Data Platforms: A Systematic Literature Review

“…In [28], the authors formally define the scheduling outcome of an executed job composed of X map tasks and Y reduce tasks. S(M apAtt ip ) is the status of a map i after the p th attempt, and S(ReduceAtt jq ) the status of reduce j after the q th attempt.…”

“…K and L represent the maximum numbers of scheduling attempts allowed for map and reduce tasks respectively. The authors [28] model the scheduling outcome of a scheduled job as follows:…”

“…AdapTive faiLure-Aware Scheduler (ATLAS ) [28] is proposed as a new scheduler for Hadoop that adapts its scheduling decisions to events occurring in the cloud environment. ATLAS can identify task failures in advance and adjust its scheduling decisions on the fly based on statistical models.…”