Astro: A predictive model for anomaly detection and feedback-based scheduling on Hadoop

Gupta, Chaitali; Bansal, Mayank; Chuang, Tzu-Cheng; Sinha, Ranjan; Ben-Romdhane, Sami

doi:10.1109/bigdata.2014.7004315

Cited by 6 publications

(3 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These values were normalized to be within the range [0,1]. In this step, we try to reduce this number by using the variance threshold technique [79] [80], i.e. using a threshold to select only those features whose variance is equal or higher.…”

Section: B Phase 2: Selection and Clusteringmentioning

confidence: 99%

KPIs-Based Clustering and Visualization of HPC Jobs: A Feature Reduction Approach

2021

View full text Add to dashboard Cite

High-Performance Computing (HPC) systems need to be constantly monitored to ensure their stability. The monitoring systems collect a tremendous amount of data about different parameters or Key Performance Indicators (KPIs), such as resource usage, IO waiting time, etc. A proper analysis of this data, usually stored as time series, can provide insight in choosing the right management strategies as well as the early detection of issues. In this paper, we introduce a methodology to cluster HPC jobs according to their KPI indicators. Our approach reduces the inherent high dimensionality of the collected data by applying two techniques to the time series: literature-based and variance-based feature extraction. We also define a procedure to visualize the obtained clusters by combining the two previous approaches and the Principal Component Analysis (PCA). Finally, we have validated our contributions on a real data set to conclude that those KPIs related to CPU usage provide the best cohesion and separation for clustering analysis and the good results of our visualization methodology. INDEX TERMS Clustering, feature extraction, high-performance computing, time series analysis.

show abstract

Section: B Phase 2: Selection and Clusteringmentioning

confidence: 99%

KPIs-Based Clustering and Visualization of HPC Jobs: A Feature Reduction Approach

2021

View full text Add to dashboard Cite

show abstract

“…In another study, Memishi et al [ 9 ] also proposed an approach that estimates the completion time of the workload and calculates the progress rate of each task to adjust the timeout value dynamically. Other studies by [ 17 , 18 , 19 , 20 ] provided predictive models based on machine learning and AI algorithms to estimate and set an optimal heartbeat timeout on the fly or to predict the failures before they occur. These approaches reduce the task fault occurrences and improve their overall performance with low latency in fault detection.…”

Section: Related Workmentioning

confidence: 99%

Experimental Analysis in Hadoop MapReduce: A Closer Look at Fault Detection and Recovery Techniques

Saadoon

Hamid

Sofian

et al. 2021

Sensors

View full text Add to dashboard Cite

Hadoop MapReduce reactively detects and recovers faults after they occur based on the static heartbeat detection and the re-execution from scratch techniques. However, these techniques lead to excessive response time penalties and inefficient resource consumption during detection and recovery. Existing fault-tolerance solutions intend to mitigate the limitations without considering critical conditions such as fail-slow faults, the impact of faults at various infrastructure levels and the relationship between the detection and recovery stages. This paper analyses the response time under two main conditions: fail-stop and fail-slow, when they manifest with node, service, and the task at runtime. In addition, we focus on the relationship between the time for detecting and recovering faults. The experimental analysis is conducted on a real Hadoop cluster comprising MapReduce, YARN and HDFS frameworks. Our analysis shows that the recovery of a single fault leads to an average of 67.6% response time penalty. Even though the detection and recovery times are well-turned, data locality and resource availability must also be considered to obtain the optimum tolerance time and the lowest penalties.

show abstract

“…In addition to the above work, Astro [101] is designed to predict anomalies in Hadoop clusters and identify the most important metrics contributing towards the failure of the scheduled tasks using different machine learning algorithms. The predictive model in Astro can detect anomalies in systems early and send a feedback to the scheduler.…”

Section: A C C E P T E D Mmentioning

confidence: 99%

Task Scheduling in Big Data Platforms: A Systematic Literature Review

Soualhia

Khomh

Tahar

2017

Journal of Systems and Software

View full text Add to dashboard Cite

Context: Hadoop, Spark, Storm, and Mesos are very well known frameworks in both research and industrial communities that allow expressing and processing distributed computations on massive amounts of data. Multiple scheduling algorithms have been proposed to ensure that short interactive jobs, large batch jobs, and guaranteed-capacity production jobs running on these frameworks can deliver results quickly while maintaining a high throughput. However, only a few works have examined the effectiveness of these algorithms. Objective: The Evidence-based Software Engineering (EBSE) paradigm and its core tool, i.e., the Systematic Literature Review (SLR), have been introduced to the Software Engineering community in 2004 to help researchers systematically and objectively gather and aggregate research evidences about different topics. In this paper, we conduct a SLR of task scheduling algorithms that have been proposed for big data platforms. Method: We analyse the design decisions of different scheduling models proposed in the literature for Hadoop, Spark, Storm, and Mesos over the period between 2005 and 2016. We provide a research taxonomy for succinct classification of these scheduling models. We also compare the algorithms in terms of performance, resources utilization, and failure recovery mechanisms. Results: Our searches identifies 586 studies from journals, conferences and workshops having the highest quality in this field. This SLR reports about different types of scheduling models (dynamic, constrained, and adaptive) and the main motivations behind them (including data locality, workload balancing, resources utilization, and energy efficiency). A discussion of some open issues and future challenges pertaining to improving the current studies is provided.

show abstract

Astro: A predictive model for anomaly detection and feedback-based scheduling on Hadoop

Cited by 6 publications

References 14 publications

KPIs-Based Clustering and Visualization of HPC Jobs: A Feature Reduction Approach

KPIs-Based Clustering and Visualization of HPC Jobs: A Feature Reduction Approach

Experimental Analysis in Hadoop MapReduce: A Closer Look at Fault Detection and Recovery Techniques

Task Scheduling in Big Data Platforms: A Systematic Literature Review

Contact Info

Product

Resources

About