LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

Abstract

Recording runtime status via logs is common for almost every computer system, and detecting anomalies in logs is crucial for timely identifying malfunctions of systems. However, manually detecting anomalies for logs is time-consuming, error-prone, and infeasible. Existing automatic log anomaly detection approaches, using indexes rather than semantics of log templates, tend to cause false alarms. In this work, we propose LogAnomaly, a framework to model unstructured a log stream as a natural language sequence. Empowered by template2vec, a novel, simple yet effective method to extract the semantic information hidden in log templates, LogAnomaly can detect both sequential and quantitive log anomalies simultaneously, which were not done by any previous work. Moreover, LogAnomaly can avoid the false alarms caused by the newly appearing log templates between periodic model retrainings. Our evaluation on two public production log datasets show that LogAnomaly outperforms existing log-based anomaly detection methods.

Using deep learning to solve computer security challenges: a survey

“…Observation 8.2: Most works in this survey adopted Phase II when parsing the raw log data. After reviewing the six works proposed recently, we found that five works (Du et al 2017;Meng et al 2019;Das et al 2018;Brown et al 2018;) employed parsing technique, while only one work (Bertero et al 2017) did not. DeepLog (Du et al 2017) parsed the raw log to different log type using Spell (Du and Li 2016) which is based a longest common subsequence.…”

“…Recently, a large number of scholars employed deep learning techniques (Du et al 2017;Meng et al 2019;Das et al 2018;Brown et al 2018;Bertero et al 2017) to detect anomaly events in the system logs and diagnosis system failures. The raw log data are unstructured, while their formats and semantics can vary significantly.…”

“…In this section, we will review the very recent six representative papers that use deep learning for system-eventbased anomaly detection (Du et al 2017;Meng et al 2019;Das et al 2018;Brown et al 2018;Bertero et al 2017). DeepLog (Du et al 2017) utilizes LSTM to model the system log as a natural language sequence, which automatically learns log patterns from the normal event, and detects anomalies when log patterns deviate from the trained model.…”

“…DeepLog (Du et al 2017) utilizes LSTM to model the system log as a natural language sequence, which automatically learns log patterns from the normal event, and detects anomalies when log patterns deviate from the trained model. LogAnom (Meng et al 2019) employs Word2vec to extract the semantic and syntax information from log templates. Moreover, it uses sequential and quantitative features simultaneously.…”

“…Bertero et al (2017) map log word to a high dimensional metric space using Google's word2vec algorithm and take it as features to classify. Among these six papers, we select two representative works (Du et al 2017;Meng et al 2019) and summarize the four phases of their approaches. We direct interested readers to Table 2 for a concise overview of these two works.…”