BERT-Log: Anomaly Detection for System Logs Based on Pre-trained Language Model

Chen, Song; Liao, Hai

doi:10.1080/08839514.2022.2145642

Cited by 34 publications

(11 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figure 3 shows how a test-taker’s actions for U01A – Party Invitations were transformed into a vector of token indices. In the final step, we ran the three vectors (token, positional, and segment embeddings) through the BERT embedding layer to form a 768-dimensional embedding vector based on the last layer (S. Chen & Liao, 2022).…”

Section: Methodsmentioning

confidence: 99%

“…In recent years, various deep learning methods have been proposed for anomaly detection in the system log files generated by an operating system, application, server, and so on (e.g., Catillo et al, 2022;Le & Zhang, 2021;Wittkopp et al, 2021). Furthermore, some researchers have combined these methods with NLP techniques to facilitate the processing of complex messages in log files (e.g., S. Chen & Liao, 2022;Guo et al, 2021;Ryciak et al, 2022;Shao et al, 2022). With pretrained large language models such as BERT (Devlin et al, 2018), sequential log entries over time or across multiple users are transformed into dense, numerical vectors (i.e., embeddings) that retain the order and dependency of the entries in a lower dimensional space.…”

Section: Anomaly Detection With Process Datamentioning

confidence: 99%

Section: Theoretical Frameworkmentioning

confidence: 99%

See 2 more Smart Citations

Unsupervised Anomaly Detection in Sequential Process Data

Bulut,

Gorgun,

2024

Zeitschrift für Psychologie

View full text Add to dashboard Cite

In this study, we present three types of unsupervised anomaly detection to identify anomalous test-takers based on their action sequences in problem-solving tasks. The first method relies on the use of the Isolation Forest algorithm to detect anomalous test-takers based on raw action sequences extracted from process data. The second method transforms raw action sequences into contextual embeddings using the Bidirectional Encoder Representations from Transformers (BERT) model and then applies the Isolation Forest algorithm to detect anomalous test-takers. The third method follows the same procedure as the second method, but it includes an intermediary step of dimensionality reduction for the contextual embeddings before applying the Isolation Forest algorithm for detecting anomalous cases. To compare the outcomes of the three methods, we analyze the log files from test-takers in the US sample ( n = 2,021) who completed the problem-solving in technology-rich environments (PSTRE) section of the Programme for the International Assessment of Adult Competencies (PIAAC) 2012 assessment. The results indicated that different groups of test-takers were flagged as anomalous depending on the representation (raw action sequences vs. contextual embeddings) and dimensionality of action sequences. Also, when the contextual embeddings were used, a larger number of test-takers were flagged by the Isolation Forest algorithm, indicating the sensitivity of this algorithm to the dimensionality of input data.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Anomaly Detection With Process Datamentioning

confidence: 99%

Section: Theoretical Frameworkmentioning

confidence: 99%

See 1 more Smart Citation

Unsupervised Anomaly Detection in Sequential Process Data

Bulut,

Gorgun,

2024

Zeitschrift für Psychologie

View full text Add to dashboard Cite

show abstract

“…These methods utilize the self-attention mechanism of the Transformer to analyze relationships within event logs, thereby improving anomaly detection accuracy. LogBERT [11]: LogBERT is a log anomaly detection method based on pre-trained language models. It draws inspiration from the BERT model used in natural language processing and applies it to event log analysis.…”

Section: Classification Of Log Anomaly Analysis Methodsmentioning

confidence: 99%

Log anomaly detection method based on CNN and LSTM fusion

Li,

Lv,

Chen

2024

Second International Conference on Informatics, Networking, and Computing (ICINC 2023)

View full text Add to dashboard Cite

Timely detection of abnormal behavior in power monitoring systems is crucial for system stability. Traditional log anomaly analysis methods have limitations, especially in handling multi-source or multi-dimensional log data. This paper introduces a method named CLSTMlog, which combines Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) networks, making the most of CNN's ability to extract local features and LSTM's capability to handle temporal relationships. Experimental results demonstrate that CLSTMlog exhibits a significant improvement in performance on the HDFS and IoT-23 datasets, including higher accuracy and reliability. This method has the potential to enhance the efficiency of log anomaly detection in power monitoring systems and overall system performance.

show abstract

“…Table III furnishes the list of state-of-art anomaly detection tools, vector generation techniques, and log features. [22] 2019   Swisslog [31] 2020    Logtransfer [32] 2020   HitAnomaly [33] 2020    LogFlow [34] 2021   Sprelog [17] 2021   LogUAD [13] 2022   DeepSyslog [26] 2022   (Event Metadata BERT-Log [35] 2022…”

Section: Comparison With Existing Techniquesmentioning

confidence: 99%

Failure Detection Using Semantic Analysis and Attention-Based Classifier Model for IT Infrastructure Log Data

Bhanage,

Pawar,

Kotecha

et al. 2023

IEEE Access

View full text Add to dashboard Cite

The improvement in the reliability, availability, and maintenance of the IT infrastructure components is paramount to ensure uninterrupted services in large-scale IT Infrastructures. The massive system logs generated by infrastructures have proved to be advantageous to pursue the run-time circumstances and behavior of the system. An existing literature has log-based failure detection techniques carrying semantic analysis but on limited log features, reflecting ineffectiveness in anomaly detection for unstable and unseen log records. We have proposed in this paper a semantic log analysis model with three log features to apprehend the gist of the log message. BERT pre-trained model is employed to adapt the feature embedding. The generated numerical vectors are further furnished to train an attention-based OLSTM (Optimized Long Short-Term Memory Networks) classifier to detect failures in diverse infrastructures. The proposed model is evaluated on five different infrastructures: Apache from a server application, OpenStack from the Distributed Systems, Windows from the Operating System, BGL from a Supercomputer, and Android from the Mobile System. The findings illustrate that the proposed system delivers improved and stable results, considering the varied IT infrastructures.

show abstract

BERT-Log: Anomaly Detection for System Logs Based on Pre-trained Language Model

Cited by 34 publications

References 39 publications

Unsupervised Anomaly Detection in Sequential Process Data

Unsupervised Anomaly Detection in Sequential Process Data

Log anomaly detection method based on CNN and LSTM fusion

Failure Detection Using Semantic Analysis and Attention-Based Classifier Model for IT Infrastructure Log Data

Contact Info

Product

Resources

About