Aarohi: Making Real-Time Node Failure Prediction Feasible

Das, A.; Mueller, Frank; Rountree, Barry

doi:10.1109/ipdps47924.2020.00115

Cited by 22 publications

(6 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Desh [18] is a deep Learning based approach obtaining high accuracy in HPC nodes failure prediction. Das et al [16] proposed Aarohi, which is an extension to Desh with higher inference performance but it still suffers as inferior failure prediction capability and long training time, because it focuses only the inference stage. Moreover, Aarohi needs re-training and re-generation if any new failure patterns occur.…”

Section: Related Workmentioning

confidence: 99%

Clairvoyant

Alharthi

Jhumka

et al. 2022

Proceedings of the 36th ACM International Conference on Supercomputing

View full text Add to dashboard Cite

System failures are expected to be frequent in the exascale era such as current Petascale systems. The health of such systems is usually determined from challenging analysis of large amounts of unstructured & redundant log data. In this paper, we leverage log data and propose Clairvoyant, a novel self-supervised (i.e., no labels needed) model to predict node failures in HPC systems based on a recent deep learning approach called transformer-decoder and the self-attention mechanism. Clairvoyant predicts node failures by (i) predicting a sequence of log events and then (ii) identifying if a failure is a part of that sequence. We carefully evaluate Clairvoyant and another state-of-the-art failure prediction approach -Desh, based on two real-world system log datasets. Experiments show that Clairvoyant is significantly better: e.g., it can predict node failures with an average Bleu, Rouge, and MCC scores of 0.90, 0.78, and 0.65 respectively while Desh scores only 0.58, 0.58, and 0.25. More importantly, this improvement is achieved with faster training and prediction time, with Clairvoyant being about 25× and 15× faster than Desh respectively.

show abstract

Section: Related Workmentioning

confidence: 99%

Clairvoyant

Alharthi

Jhumka

et al. 2022

Proceedings of the 36th ACM International Conference on Supercomputing

View full text Add to dashboard Cite

show abstract

“…This plan makes areas of strength for a standard with an unsettling influence dismissal term for every specialist, trailed by a further developed MAS control framework with no aggravation term. Making a time span for Node Failure Prediction is accessible with Aarohi [6]. Aarohi is a structure that offers a successful method for estimating disappointments on the web.…”

Section: Related Workmentioning

confidence: 99%

Mapping of Virtual Machines Using Machine Learning Algorithms for Detection of Faulty Nodes

Gaykar¹,

Khanaa²,

Joshi³

2022

IJSSE

View full text Add to dashboard Cite

A distributed system is characterized by a large number of nodes that are linked to a network and are mostly used for transaction processing. Large set of users are likely to communicate information over the network to the nodes, consistency and dependability remain a critical problem in the distributed environments. Independent failure of the component is one of the major problems in the distributed systems as it slowly impacts the performance of the other nodes in the system. The quality of service - QoS of a distributed network may be improved by a quick way of detecting problematic nodes. Sometime heavy nodes required high computation for transaction processing while idle nodes take low computation. In this paper, we proposed identification of straggler nodes in distributed environment with the help of hybrid machine learning algorithm. The work basically carried out to set up of large number of virtual machines and collect current log audits of each VM. According to the available parameters of audit files to each machine, algorithms decide that specific node is overheated or ideal condition. In expensive experimental analysis we demonstrate a accuracy of proposed hybrid machine learning algorithm. The proposed algorithm produces higher precision up to 4.5% than state-of-art methods. Key highlights of the VM mapping strategy were also investigated through a scrutiny of ongoing contracts. Main focus remains on machine learning (ML) to distinguish PM (Physical Machine) congestion, determining VMs from crowded PMs, and VM conditions as major exercises. This paper aims to review and characterize research on the planning and status of VMs that use ML using asset usage history. Energy productivity, VM migration, and quality of service were the main exhibition boundaries used to investigate cloud data center presentations.

show abstract

“…(2) Then it re-trains chain recognition of events augmented with expected lead times to failure; (3) Finally Desh predicts lead times during testing/inference deployment to predict which specific node fails in how many minutes. To speedup the recognition of the failure chain from log events, Das et al [34] proposed a new node failure predicting method called Aarohi to extend their previous work to online learning setting. Aarohi first trains an offline deep learning model with log parsing, then utilizes grammar-based rules to provide online testing.…”

Section: Homogeneous Systemsmentioning

confidence: 99%

A Survey on Automated Log Analysis for Reliability Engineering

Chen

et al. 2020

Preprint

View full text Add to dashboard Cite

Logs are semi-structured text generated by logging statements in software source code. In recent decades, software logs have become imperative in the reliability assurance mechanism of many software systems because they are often the only data available that record software runtime information. As modern software is evolving into a large scale, the volume of logs has increased rapidly. To enable effective and efficient usage of modern software logs in reliability engineering, a number of studies have been conducted on automated log analysis. This survey presents a detailed overview of automated log analysis research, including how to automate and assist the writing of logging statements, how to compress logs, how to parse logs into structured event templates, and how to employ logs to detect anomalies, predict failures, and facilitate diagnosis. Additionally, we survey work that releases open-source toolkits and datasets. Based on the discussion of the recent advances, we present several promising future directions toward real-world and next-generation automated log analysis.CCS Concepts: • Software and its engineering → Software maintenance tools; Software creation and management.

show abstract

Aarohi: Making Real-Time Node Failure Prediction Feasible

Cited by 22 publications

References 36 publications

Clairvoyant

Clairvoyant

Mapping of Virtual Machines Using Machine Learning Algorithms for Detection of Faulty Nodes

A Survey on Automated Log Analysis for Reliability Engineering

Contact Info

Product

Resources

About