An online log template extraction method based on hierarchical clustering

Yang, Ruipeng; Qu, Dan; Qian, Yekui; Dai, Yusheng; Zhu, Shaowei

doi:10.1186/s13638-019-1430-4

Cited by 13 publications

(4 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A log sequence can be considered as a series of events that occur, namely, the sequence of the log template corresponding to the original log sequence. The extraction of log templates from the original log has been studied in literature [35]. The study work in this paper is to detect the anomaly of the log template sequence corresponding to the original log sequence.…”

Section: Nlsalog Frameworkmentioning

confidence: 99%

nLSALog: An Anomaly Detection Framework for Log Sequence in Security Management

Yang

Gao

et al. 2019

IEEE Access

Self Cite

View full text Add to dashboard Cite

For the security defense in the current Intelligent Transportation System (ITS), malware is often used as the security analysis data source, but only the known attack type can be detected. A general anomaly detection framework is proposed, using log data as the analysis data source. By modeling the log template sequence as a natural language sequence and using the stacked Long Short-Term Memory (LSTM) with self-attention mechanism, the framework can effectively extract the hidden pattern of the log template sequence, and well express the dependencies inside the log template sequence. The experimental results show that the overall accuracy of log sequence anomaly detection of the detection framework is better than that of existing methods and the time cost is lower.

show abstract

Section: Nlsalog Frameworkmentioning

confidence: 99%

nLSALog: An Anomaly Detection Framework for Log Sequence in Security Management

Yang

Gao

et al. 2019

IEEE Access

Self Cite

View full text Add to dashboard Cite

show abstract

“…This study compiles available evaluation results using these metrics for method comparisons. Although Dendrogram purity [35], Levenshtein edit distance [226], and loss functions [34] also appear, they are used infrequently. Also, there are a number of studies that use a stricter form of PA (i.e., requiring all dynamic parameters to be identified for the template to be considered correctly parsed) [30] [194].…”

Section: B Other Performance Metricsmentioning

confidence: 99%

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

Lupton,

Washizaki,

Yoshioka

et al. 2024

IEEE Access

View full text Add to dashboard Cite

As production system estates become larger and more complex, ensuring stability through traditional monitoring approaches becomes more challenging. Rule-based monitoring is common in industrial settings, but it has limitations. These include the difficulty of crafting rules capable of detecting unforeseen issues and the burden of manually maintaining rule sets. A potential solution to effectively manage complex system states is log anomaly detection. Workflows for log anomaly detection utilize several fundamental components. These include preprocessors for data cleansing, parsers to extract structured information from raw log data, encoding algorithms to convert extracted data into usable model input features, anomaly detection methods to isolate anomalous signals, and feedback mechanisms to incrementally improve model performance. This study explores the current state of research into online parser-supported log anomaly detection methods, investigates recent research trends, compares the performances of parser and anomaly detection methods using common public datasets and metrics, and assesses their performance evolution over time. Additionally, it classifies available methods using a newly introduced taxonomy, highlights current research gaps, and recommends future research directions.INDEX TERMS Log parsing, log template extraction, online algorithms, anomaly detection.

show abstract

“…Xu et al [5] utilized PCA to identify abnormal events and visualize the final results. Yang et al [35] proposed LogOHC, which has high extraction efficiency in multi-source log datasets. Unsupervised learning with high automation saves labor costs, but most detection uses clustering and correlation analysis, and the degree of automated correlation between these methods is not high.…”

Section: Anomaly Detectionmentioning

confidence: 99%

LTAnomaly: A Transformer Variant for Syslog Anomaly Detection Based on Multi-Scale Representation and Long Sequence Capture

Han

Sun

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

Detailed information on system operation is recorded by system logs, from which fast and accurate detection of anomalies is conducive to service management and system maintenance. Log anomaly detection methods often only handle a single type of anomaly, and the utilization of log messages could be higher, which makes it challenging to improve the performance of log anomaly detection models. This article presents the LTAnomaly model to accomplish log anomaly detection using semantic information, sequence relationships, and component values to make a vector representation of logs, and we add Transformer with long short-term memory (LSTM) as our final classification model. When sequences are processed sequentially, the model is also influenced by the information from the global information, thus increasing the dependence on feature information. This improves the utilization of log messages with a flexible, simple, and robust model. To evaluate the effectiveness of our method, experiments are performed on the HDFS and BGL datasets, with the F1-measures reaching 0.985 and 0.975, respectively, showing that the proposed method enjoys higher accuracy and a more comprehensive application range than existing models.

show abstract

An online log template extraction method based on hierarchical clustering

Cited by 13 publications

References 32 publications

nLSALog: An Anomaly Detection Framework for Log Sequence in Security Management

nLSALog: An Anomaly Detection Framework for Log Sequence in Security Management

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

LTAnomaly: A Transformer Variant for Syslog Anomaly Detection Based on Multi-Scale Representation and Long Sequence Capture

Contact Info

Product

Resources

About