Pinpointing Anomaly Events in Logs from Stability Testing -- N-Grams vs. Deep-Learning

Abstract

As stability testing execution logs can be very long, software engineers need help in locating anomalous events. We develop and evaluate two models for scoring individual logevents for anomalousness, namely an N-Gram model and a Deep Learning model with LSTM (Long short-term memory). Both are trained on normal log sequences only. We evaluate the models with long log sequences of Android stability testing in our company case and with short log sequences from HDFS (Hadoop Distributed File System) public dataset. We evaluate next event prediction accuracy and computational efficiency. The LSTM model is more accurate in stability testing logs (0.848 vs 0.865), whereas in HDFS logs the N-Gram is slightly more accurate (0.904 vs 0.900). The N-Gram model has far superior computational efficiency compared to the Deep model (4 to 13 seconds vs 16 minutes to nearly 4 hours), making it the preferred choice for our case company. Scoring individual log events for anomalousness seems like a good aid for root cause analysis of failing test cases, and our case company plans to add it to its online services. Despite the recent surge in using deep learning in software system anomaly detection, we found limited benefits in doing so. However, future work should consider whether our finding holds with different LSTM-model hyper-parameters, other datasets, and with other deep-learning approaches that promise better accuracy and computational efficiency than LSTM based models.

How to Configure Masked Event Anomaly Detection on Software Logs?

“…As in prior work, we reason that incorrectness or low scores in event predictions in anomalous sequences are good candidates for true anomalous events. This paper extends prior work [2] by investigating more models, datasets, and settings. For example, this study introduces CNN as another DL model next to LSTM as it has proven to be an effective alternative for anomaly detection [5], [6].…”

“…Mäntylä et al [2] utilized LSTM and N-Gram models to pinpoint anomalous events in log files. The study focused on the company Profilence that provides test automation and telemetry solutions.…”

“…Log event prediction is one of the strategies of anomaly detection providing means for root cause detection as well as failure identification, tolerance, and recovery [1]. There is existing work that has focused on event prediction for anomaly detection e.g., [2]- [4]. Pinpointing anomalous events can help test and application engineers to identify and fix subtle bugs that result in crashes after a long time [2].…”

“…There is existing work that has focused on event prediction for anomaly detection e.g., [2]- [4]. Pinpointing anomalous events can help test and application engineers to identify and fix subtle bugs that result in crashes after a long time [2].…”

“…Existing work shows that event prediction can provide scores for log events of a software log under investigation [2]. These scores represent event suspiciousness or anomalousness and may help software engineers that are going through a log file that can be tens of thousands of lines long.…”