An Anomaly Detection Method for System Logs Using Venn-Abers Predictors

Pan, Lanlan; Gu, Zhaojun; Ren, Yitong; Liu, Chunbo; Wang, Zhi

doi:10.1109/dsc50466.2020.00063

Cited by 5 publications

(3 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Venn-ABERS predictors have been successfully applied in different fields, such as drug discovery (Buendia et al, 2019), compound activity prediction (Toccaceli et al, 2016), adversarial manipulation detection (Peck et al, 2020) and log anomaly detection (Pan et al, 2020). To the best of our knowledge, we are the first to apply Venn-ABERS prediction to modelling uncertainty in NLU.…”

Section: Related Workmentioning

confidence: 99%

Calibration of Natural Language Understanding Models with Venn--ABERS Predictors

Giovannotti¹

2022

Preprint

View full text Add to dashboard Cite

Transformers, currently the state-of-the-art in natural language understanding (NLU) tasks, are prone to generate uncalibrated predictions or extreme probabilities, making the process of taking different decisions based on their output relatively difficult. In this paper we propose to build several inductive Venn-ABERS predictors (IVAP), which are guaranteed to be well calibrated under minimal assumptions, based on a selection of pre-trained transformers. We test their performance over a set of diverse NLU tasks and show that they are capable of producing well-calibrated probabilistic predictions that are uniformly spread over the [0,1] interval -all while retaining the original model's predictive accuracy.

show abstract

Section: Related Workmentioning

confidence: 99%

Calibration of Natural Language Understanding Models with Venn--ABERS Predictors

Giovannotti¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…By this way it is convenient to use the loss function to compare the effectiveness of various algorithms. The experimental results turn out that the method of using Venn-Abers predictors to evaluate the correctness of the system anomaly detection is valid and accurate [27]. Moreover, considering the differences in the data processing principles of different algorithms, full play is given to the advantages of each model, and integrated learning is used in order to achieve a stronger generalization ability.…”

Section: Introductionmentioning

confidence: 98%

Valid Probabilistic Anomaly Detection Models for System Logs

Liu

Pan

et al. 2020

Wireless Communications and Mobile Computing

Self Cite

View full text Add to dashboard Cite

System logs can record the system status and important events during system operation in detail. Detecting anomalies in the system logs is a common method for modern large-scale distributed systems. Yet threshold-based classification models used for anomaly detection output only two values: normal or abnormal, which lacks probability of estimating whether the prediction results are correct. In this paper, a statistical learning algorithm Venn-Abers predictor is adopted to evaluate the confidence of prediction results in the field of system log anomaly detection. It is able to calculate the probability distribution of labels for a set of samples and provide a quality assessment of predictive labels to some extent. Two Venn-Abers predictors LR-VA and SVM-VA have been implemented based on Logistic Regression and Support Vector Machine, respectively. Then, the differences among different algorithms are considered so as to build a multimodel fusion algorithm by Stacking. And then a Venn-Abers predictor based on the Stacking algorithm called Stacking-VA is implemented. The performances of four types of algorithms (unimodel, Venn-Abers predictor based on unimodel, multimodel, and Venn-Abers predictor based on multimodel) are compared in terms of validity and accuracy. Experiments are carried out on a log dataset of the Hadoop Distributed File System (HDFS). For the comparative experiments on unimodels, the results show that the validities of LR-VA and SVM-VA are better than those of the two corresponding underlying models. Compared with the underlying model, the accuracy of the SVM-VA predictor is better than that of LR-VA predictor, and more significantly, the recall rate increases from 81% to 94%. In the case of experiments on multiple models, the algorithm based on Stacking multimodel fusion is significantly superior to the underlying classifier. The average accuracy of Stacking-VA is larger than 0.95, which is more stable than the prediction results of LR-VA and SVM-VA. Experimental results show that the Venn-Abers predictor is a flexible tool that can make accurate and valid probability predictions in the field of system log anomaly detection.

show abstract

“…different methods can be used for scheduling. existing algorithms can be divided into two classes, online mode and categories [2]. in the online mode, a task is assigned to the appropriate resource or machine (in terms of timing and strategy) as soon as the grid scheduler is reached; but in the case of batch tasks, they are not scheduled immediately, but become a group (set of tasks) and with a specific time period, the scheduling operation is performed on them, in fact, the scheduling operation is done in groups and categories.…”

Section: Introductionmentioning

confidence: 99%

Development of an Anomaly Detection Algorithm in Distributed Systems based on Runtime and System Status

Asgarnezhad

alwali

Alsaedi

et al. 2022

WJCMS

View full text Add to dashboard Cite

Computational grades have emerged as a new approach to solving large-scale problems in the fields of science, engineering, and commerce. The computing grid is a hardware and software infrastructure that provides affordable, reliable, comprehensive, and affordable access to the computational abilities of others. A computational grid is associated with a set of resources on a large scale. Computational grades have emerged as a new approach to solving large-scale problems in the fields of science, engineering, and commerce. The computing grid is a hardware and software infrastructure that provides affordable, reliable, comprehensive, and affordable access to the computational abilities of others. A computational grid is associated with a set of resources on a large scale. The purpose of this thesis is to provide a method that optimizes resource management and scheduling in at least one direction. The main focus of the research is on the time criterion, the deadline for doing things and receiving the response from the grid are parameters that can be examined. From a more general perspective, the aim of the research is to get the answer as quickly as possible from the calculation grid. The proposed algorithm improves the scheduling and resource management of the grid in the direction of improvement, and the structure and form of this problem have not yet been resolved. The proposed solution has been considered hypothesis and removed some of the definitions of grid scheduling, such as cost, quality of service, architectures, and others, but ultimately heed it and timed the grid in some ways.

show abstract

An Anomaly Detection Method for System Logs Using Venn-Abers Predictors

Cited by 5 publications

References 14 publications

Calibration of Natural Language Understanding Models with Venn--ABERS Predictors

Calibration of Natural Language Understanding Models with Venn--ABERS Predictors

Valid Probabilistic Anomaly Detection Models for System Logs

Development of an Anomaly Detection Algorithm in Distributed Systems based on Runtime and System Status

Contact Info

Product

Resources

About