Predicting Scheduling Failures in the Cloud: A Case Study with Google Clusters and Hadoop on Amazon EMR

Soualhia, Mbarka; Khomh, Foutse; Tahar, Sofiène

doi:10.1109/hpcc-css-icess.2015.170

Cited by 24 publications

(16 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A failure prediction model has been proposed in our previous work [42], and we improved the model's accuracy by using multiple feature selection techniques. Our failure prediction model outperformed others in previous studies [8,33]. Table 1 summarizes the most current studies on the analysis and prediction of job and task failures.…”

Section: Failure Predictionmentioning

confidence: 73%

“…Earlier research on job failure has mostly focused on the study and characterization of failures. However, little research has been published on the prediction of job/task failure [1,8,10,[31][32][33][34]. Samak et al [35] have applied the Naive Bayes classification algorithm to the execution logs of scientific processes to predict the failure of tasks.…”

Section: Failure Predictionmentioning

confidence: 99%

See 1 more Smart Citation

Analysis of Job Failure and Prediction Model for Cloud Computing Using Machine Learning

Jassas

Mahmoud

2022

Sensors

View full text Add to dashboard Cite

Modern applications, such as smart cities, home automation, and eHealth, demand a new approach to improve cloud application dependability and availability. Due to the enormous scope and diversity of the cloud environment, most cloud services, including hardware and software, have encountered failures. In this study, we first analyze and characterize the behaviour of failed and completed jobs using publicly accessible traces. We have designed and developed a failure prediction model to determine failed jobs before they occur. The proposed model aims to enhance resource consumption and cloud application efficiency. Based on three publicly available traces: the Google cluster, Mustang, and Trinity, we evaluate the proposed model. In addition, the traces were also subjected to various machine learning models to find the most accurate one. Our results indicate a significant correlation between unsuccessful tasks and requested resources. The evaluation results also revealed that our model has high precision, recall, and F1-score. Several solutions, such as predicting job failure, developing scheduling algorithms, changing priority policies, or limiting re-submission of tasks, can improve the reliability and availability of cloud services.

show abstract

Section: Failure Predictionmentioning

confidence: 73%

Section: Failure Predictionmentioning

confidence: 99%

Analysis of Job Failure and Prediction Model for Cloud Computing Using Machine Learning

Jassas

Mahmoud

2022

Sensors

View full text Add to dashboard Cite

show abstract

“…RNN achieved an accuracy of around 84%. Soualhia et al [13] explored possibility of predicting application task failure in cloud platform to enhance performance of resources that are used to execute tasks. The authors applied a set of statistical and machine learning models such as Decision Tree (DT), Boost, and Random Forest (RF) to predict task failure.…”

Section: A Task Failure Predictionmentioning

confidence: 99%

Proactive Failure-Aware Task Scheduling Framework for Cloud Computing

2021

View full text Add to dashboard Cite

Cloud computing is a widely adopted platform for executing tasks of different application types that belong to the end users. In the cloud, application task is prone to failure for several reasons, such as software bug or exception, virtual or physical infrastructure failure. Cloud service providers are responsible for managing availability of scheduled computing tasks in order to provide high level QoS for their customers. Protecting task against failure is a challenging and not a trivial mission due to dynamic, heterogeneous and large distributed structure of the cloud environment. The existing works in the literature focus on task failure prediction and neglect the remedy (post) actions. In this work, we first study and analyze three publicly available large cluster datasets from Google, Alibaba, and Trinity, to characterize task failure in cloud computing platform. We then propose a failure-aware task scheduling framework that can predict the termination status for a set of given tasks during the runtime, and take the appropriate remedy actions. The framework uses deep learning methods named Artificial and Convolutional Neural Network, ANN and CNN, for different prediction purposes. In addition, we formalize the actions selection problem as Integer Linear Programming (ILP) model and propose a heuristic optimization solution that aims to minimize the failure probability of tasks and their resources usage. The results show ANN and CNN can achieve prediction accuracy of up to 94% and 92%, respectively using Google dataset. Moreover, the framework can protect up to 40% of tasks that are predicted as failed using Alibaba dataset by taking the appropriate remedy actions, and hence save many of cluster's resources such as CPU and RAM.INDEX TERMS Task failure prediction, deep learning, task scheduling, cloud computing.

show abstract

“…The public Google Traces provides data about task and job failures in real world Google clusters. We found that more than 40% of the tasks and jobs can be failed [7]. Therefore, in our case study, we performed different simulations of varying the injected failure rates, with a maximum failure rate of 40%.…”

Section: Setup Of the Case Studymentioning

confidence: 99%

“…In addition, we used AnarchyApe [18] to inject different types of failures in Hadoop machines. We relied on rate of failures observed in Google clusters [7]. It is possible that the majority of Hadoop clusters do not experience such high rate of failures.…”

Section: Threats To Validitymentioning

confidence: 99%

ATLAS: An AdapTive faiLure-Aware Scheduler for Hadoop

Soualhia

Khomh

Tahar

2015

2015 IEEE 34th International Performance Computing and Communications Conference (IPCCC)

Self Cite

View full text Add to dashboard Cite

Hadoop has become the de facto standard for processing large data in today's cloud environment. The performance of Hadoop in the cloud has a direct impact on many important applications ranging from web analytic, web indexing, image and document processing to high-performance scientific computing. However, because of the scale, complexity and dynamic nature of the cloud, failures are common and these failures often impact the performance of jobs running in Hadoop. Although Hadoop possesses built-in failure detection and recovery mechanisms, several scheduled jobs still fail because of unforeseen events in the cloud environment. A single task failure can cause the failure of the whole job and unpredictable job running times. In this paper, we propose ATLAS (AdapTive faiLure-Aware Scheduler), a new scheduler for Hadoop that can adapt its scheduling decisions to events occurring in the cloud environment. Using statistical models, ATLAS predicts task failures and adjusts its scheduling decisions on the fly to reduce task failure occurrences. We implement ATLAS in the Hadoop framework of Amazon Elastic MapReduce (EMR) and perform a case study to compare its performance with those of the FIFO, Fair and Capacity schedulers. Results show that ATLAS can reduce the percentage of failed jobs by up to 28% and the percentage of failed tasks by up to 39%, and the total execution time of jobs by 10 minutes on average. ATLAS also reduces CPU and memory usages.

show abstract

Predicting Scheduling Failures in the Cloud: A Case Study with Google Clusters and Hadoop on Amazon EMR

Cited by 24 publications

References 11 publications

Analysis of Job Failure and Prediction Model for Cloud Computing Using Machine Learning

Analysis of Job Failure and Prediction Model for Cloud Computing Using Machine Learning

Proactive Failure-Aware Task Scheduling Framework for Cloud Computing

ATLAS: An AdapTive faiLure-Aware Scheduler for Hadoop

Contact Info

Product

Resources

About