Deep Learning for Anomaly Detection in Log Data: A Survey

Abstract

Automatic log file analysis enables early detection of relevant incidents such as system failures. In particular, selflearning anomaly detection techniques capture patterns in log data and subsequently report unexpected log event occurrences to system operators without the need to provide or manually model anomalous scenarios in advance. Recently, an increasing number of approaches leveraging deep learning neural networks for this purpose have been presented. These approaches have demonstrated superior detection performance in comparison to conventional machine learning techniques and simultaneously resolve issues with unstable data formats. However, there exist many different architectures for deep learning and it is nontrivial to encode raw and unstructured log data to be analyzed by neural networks. We therefore carry out a systematic literature review that provides an overview of deployed models, data pre-processing mechanisms, anomaly detection techniques, and evaluations. The survey does not quantitatively compare existing approaches but instead aims to help readers understand relevant aspects of different model architectures and emphasizes open issues for future work.

LogGD:Detecting Anomalies from System Logs by Graph Neural Networks

“…In our experiments, four public log datasets, HDFS, BGL, Spirit and Thunderbird, are used to evaluate the proposed approach and the relevant baseline methods. The datasets are widely used in log analysis research [1]- [3], [5], [45], [46] because all of them come from real-world datasets and are labeled either manually by system administrators or through alert tags automatically generated by their systems. We obtained all the log datasets from the publicly available websites.…”

“…Also, LogGD identifies anomalies at the graph level. Developers and operators may have to inspect each event in the data window to locate the potential fault [46]. It would be interesting to explore the feasibility of more fine-grained anomaly detection to reduce the effort and time to locate a fault.…”

Detecting and Processing Anomalies in a Factory of the Future

“…Anomalies such as unplanned production downtimes cost industrial manufacturers around $50 billion per year [27], and real-world examples for anomalies caused by cyber-attacks with financial losses for the victims are given in Sections 2.1 and 2.2. Hence, it is not surprising that the timely detection of anomalies is already the focus of a large number of research activities [26,[28][29][30]. As already stated in the Introduction, the high dynamics and the high level of connectivity between production systems offer new potential for optimized production but lead at the same time to new challenges in anomaly detection, such as the problem of detecting anomalies in frequently changing factory situations and the detection of cyber-attacks in addition to anomalies in system and process behavior.…”