Log-based anomaly detection with deep learning

Le, Van-Hoang; Zhang, Hongyu

doi:10.1145/3510003.3510155

Cited by 103 publications

(46 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The dataset contains suitably labeled normal and alert events, with the alert events considered as anomalies. The BGL dataset does not have a session ID with which to group the data, thus the dataset is chunked into log paragraphs, each consisting of the consecutive log sentences belonging to a time window of 30 seconds, following the approach discussed in [5], [7]. A log paragraph is considered an anomaly if it includes at least one log sentence that is tagged as an anomalous event.…”

Section: A Experimental Setupmentioning

confidence: 99%

“…The dataset has also been labeled in the same manner as the BGL data set. As with the BGL dataset, the Thunderbird dataset does not have a session ID, so the log sentences are similarly chunked into paragraphs, each consisting of the consecutive sentences belonging to a time window of 30 seconds, following the approach discussed in [5], [7]. Furthermore, a log paragraph is considered an anomaly if it includes at least one log sentence that is tagged as an anomalous event.…”

Section: A Experimental Setupmentioning

confidence: 99%

“…A model with high Specificity can correctly identify normal samples, which makes it more likely that any detected anomalies are genuine. Furthermore, Le and Zhang [7] have noted that a high Specificity can help mitigate the impact of having an imbalanced class distribution on the model's overall performance.…”

Section: A Experimental Setupmentioning

confidence: 99%

“…Due to the high cost of preparing labeled data, classificationbased methods such as LogSy [6] have limited utility in production settings. As a result, the majority of techniques used in detecting log anomalies assume a zero-positive training scenario, where normal log data is the only data available for self-supervised training [7] only. Furthermore, [8] considers two categories of self-supervised models for log anomaly detection.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

LogFiT: Log Anomaly Detection using Fine-Tuned Language Models

Almodovar¹,

Sabrina²,

Karimi³

et al. 2023

Preprint

View full text Add to dashboard Cite

<p>System logs are a valuable source of information for monitoring and maintaining the security and stability of computer systems. Techniques based on Deep Learning and Natural Language Processing have demonstrated effectiveness in detecting abnormal behavior from these system logs. However existing approaches are inflexible and impractical: techniques that rely on log templates are unable to handle variability in log content, while classification-based approaches require labeled data for supervised training. In this paper, a novel log anomaly detection model named LogFiT is proposed. The LogFiT model is robust to changes in log content and only requires self-supervised training. The LogFiT model uses a pretrained BERT-based language model fine-tuned to recognise the linguistic patterns of the normal log data. The LogFiT model is trained using masked token prediction on the normal log data only. Consequently when presented with the new log data, the model's top-k token prediction accuracy is used as threshold for determining whether the new log data has deviated from the normal log data. Experimental results show that LogFiT's F1 score exceeds that of baselines on the HDFS, BGL and Thunderbird datasets. Critically, when variability in the log data is introduced during evaluation, LogFiT's effectiveness surpasses that of the baseline models.</p>

show abstract

Section: A Experimental Setupmentioning

confidence: 99%

Section: A Experimental Setupmentioning

confidence: 99%

Section: A Experimental Setupmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

LogFiT: Log Anomaly Detection using Fine-Tuned Language Models

Almodovar¹,

Sabrina²,

Karimi³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Log Analysis with Language Models: Log analysis is a research area that has attracted lots of attention due to its practical importance. Typical applications of log analysis include anomaly detection [1], [54], [55], [56], failure prediction [7], [8], root cause analysis [5], [6], etc. Recently, inspired by the success of pre-trained models in NLP, many studies have been proposed to apply pre-trained language models to log analysis.…”

Section: Related Workmentioning

confidence: 99%

Log-based Anomaly Detection Without Log Parsing

Zhang

2021

2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)

Self Cite

123

View full text Add to dashboard Cite

Logs generated by large-scale software systems provide crucial information for engineers to understand the system status and diagnose problems of the systems. Log parsing, which converts raw log messages into structured data, is the first step to enabling automated log analytics. Existing log parsers extract the common part as log templates using statistical features. However, these log parsers often fail to identify the correct templates and parameters because: 1) they often overlook the semantic meaning of log messages, and 2) they require domain-specific knowledge for different log datasets. To address the limitations of existing methods, in this paper, we propose LogPPT to capture the patterns of templates using prompt-based few-shot learning. LogPPT utilises a novel prompt tuning method to recognise keywords and parameters based on a few labelled log data. In addition, an adaptive random sampling algorithm is designed to select a small yet diverse training set. We have conducted extensive experiments on 16 public log datasets. The experimental results show that LogPPT is effective and efficient for log parsing.

show abstract

Assessing the impact of bag‐of‐words versus word‐to‐vector embedding methods and dimension reduction on anomaly detection from log files

Qiu,

Zhou,

Niblett

et al. 2023

Int J Network Mgmt

View full text Add to dashboard Cite

In terms of cyber security, log files represent a rich source of information regarding the state of a computer service/system. Automating the process of summarizing log file content represents an important aid for decision‐making, especially given the 24/7 nature of network/service operations. We perform benchmarking over eight distinct log files in order to assess the impact of the following: (1) different embedding methods for developing semantic descriptions of the original log files, (2) applying dimension reduction to the high‐dimensional semantic space, and (3) assessing the impact of using different unsupervised learning algorithms for providing a visual summary of the service state. Benchmarking demonstrates that (1) word‐to‐vector embeddings identified by bidirectional encoder representation from transformers (BERT) without “fine‐tuning” are sufficient to match the performance of Bag‐or‐Words embeddings provided by term frequency‐inverse document frequency (TF‐IDF) and (2) the self‐organizing map without dimension reduction provides the most effective anomaly detector.

show abstract

Log-based anomaly detection with deep learning

Cited by 103 publications

References 29 publications

LogFiT: Log Anomaly Detection using Fine-Tuned Language Models

LogFiT: Log Anomaly Detection using Fine-Tuned Language Models

Log-based Anomaly Detection Without Log Parsing

Assessing the impact of bag‐of‐words versus word‐to‐vector embedding methods and dimension reduction on anomaly detection from log files

Contact Info

Product

Resources

About