LogTransfer: Cross-System Log Anomaly Detection for Software Systems with Transfer Learning

Chen, Rui; Zhang, Shenglin; Li, Dongwen; Zhang, Yuzhe; Guo, Fangrui; Meng, Weibin; Pei, Dan; Zhang, Yuzhi; Xu, Chuang; Liu, Yuqing

doi:10.1109/issre5003.2020.00013

Cited by 61 publications

(24 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These LSTM-based methods encountered major performance issues when processing unseen log events and values. Chen et al 38 addressed this problem by training LSTM models and introducing a novel transfer learning method. Unfortunately, they transferred only the anomalous knowledge, not the actual parsing of values.…”

Section: Related Workmentioning

confidence: 99%

FlexParser—The adaptive log file parser for continuous results in a changing world

Rücker

Maier

2022

J Software Evolu Process

View full text Add to dashboard Cite

Any modern system writes events into files, called log files. Those contain crucial information which are subject to various analyses. Examples range from cybersecurity, intrusion detection over usage analyses to trouble shooting. Before data analysis is possible, desired information needs to be extracted first out of the semi-structured log messages. State-of-the-art event parsing often assumes static log events. However, any modern system is updated consistently and with updates also log file structures can change. We call those changes "mutation" and study parsing performance for different mutation cases. Latest research discovers mutations using anomaly detection post mortem, however, does not cover actual continuous parsing. Thus, we propose a novel and flexible parser, called FlexParser, which can extract desired values despite gradual changes in the log messages. It implies basic text preprocessing followed by a supervised Deep Learning method. We train a stateful LSTM on parsing one event per data set. Statefulness enforces the model to learn log message structures across several examples. Our model was tested on seven different, publicly available log file data sets and various kinds of mutations. Exhibiting an average F1-Score of 0.98, it outperforms other Deep Learning methods as well as state-of-the-art unsupervised parsers.

show abstract

Section: Related Workmentioning

confidence: 99%

FlexParser—The adaptive log file parser for continuous results in a changing world

Rücker

Maier

2022

J Software Evolu Process

View full text Add to dashboard Cite

show abstract

“…Then, we evaluate our method on the Hadoop dataset by finding anomalous samples in the Hadoop. We compare our method with LogTransfer [45], a log-based domain adaptation method. The LogTransfer is trained on the HDFS dataset and then evaluate on the Hadoop dataset.…”

Section: Does Transformer Really Benefit From More Unstructuredmentioning

confidence: 99%

UniLog: Deploy One Model and Specialize it for All Log Analysis Tasks

Zhu¹,

Meng²,

Liu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Log analysis is vitally important for network reliability, and many log-based tasks are derived to analysis logs, such as anomaly detection, failure prediction, log compression, and log summarization. It is desired to have a unified log analysis framework to simultaneously run all these log analysis tasks on one model to achieve deployment convenience, superior task performance, and low maintenance cost. However, due to severe challenges about log data heterogeneity across devices, pioneer works design specialized algorithms for each task. In this work, we formulate the log analysis as a multi-task learning approach and propose to train a single model that can perform various log analysis tasks. We name this unified log analysis approach as UniLog. To effectively build an UniLog model, we propose a log data pretrained transformer to utilize the enormous unlabeled log data, and a corresponding multi-log-tasking finetune strategy for various log analysis tasks. Extensive experiments across seven datasets on four log analysis tasks demonstrate that UniLog achieves remarkable performance.

show abstract

“…Chen et al [98] -Proposes LogTransfer, to transfer anomalous log knowledge from the source system to the target system.…”

Section: Du Et Al [5] -Presentsmentioning

confidence: 99%

“…2020 TNSM (J) LogSayer: Log pattern-driven cloud component anomaly diagnosis with machine learning [138]. 2020 IWQoS (C) (J) LogTransfer: Cross-system log anomaly detection for software systems with transfer learning [98]. 2020 ISSRE (C) Semi-supervised log-based anomaly detection via probabilistic label estimation [89].…”

Section: List Of Papersmentioning

confidence: 99%

A Comprehensive Survey of Logging in Software: From Logging Statements Automation to Log Mining and Analysis

Gholamian¹,

Ward²

2021

Preprint

View full text Add to dashboard Cite

Logs are widely used to record runtime information of software systems, such as the timestamp and the importance of an event, the unique ID of the source of the log, and a part of the state of a task's execution. The rich information of logs enables system developers (and operators) to monitor the runtime behaviors of their systems and further track down system problems and perform analysis on log data in production settings. However, the prior research on utilizing logs is scattered and that limits the ability of new researchers in this field to quickly get to the speed and hampers currently active researchers to advance this field further. Therefore, this paper surveys and provides a systematic literature review and mapping of the contemporary logging practices and log statements' mining and monitoring techniques and their applications such as in system failure detection and diagnosis. We study a large number of conference and journal papers that appeared on top-level peer-reviewed venues. Additionally, we draw high-level trends of ongoing research and categorize publications into subdivisions. In the end, and based on our holistic observations during this survey, we provide a set of challenges and opportunities that will lead the researchers in academia and industry in moving the field forward.

show abstract

LogTransfer: Cross-System Log Anomaly Detection for Software Systems with Transfer Learning

Cited by 61 publications

References 29 publications

FlexParser—The adaptive log file parser for continuous results in a changing world

FlexParser—The adaptive log file parser for continuous results in a changing world

UniLog: Deploy One Model and Specialize it for All Log Analysis Tasks

A Comprehensive Survey of Logging in Software: From Logging Statements Automation to Log Mining and Analysis

Contact Info

Product

Resources

About