Online Anomaly Detection by Improved Grammar Compression of Log Sequences

Abstract

Nowadays, log sequences mining techniques are widely used in detecting anomalies for Internet services. The state-ofthe-art anomaly detection methods either need significant computational costs, or require specific assumptions that the test logs are holding certain data distribution patterns in order to be effective. Therefore, it is very difficult to achieve real time responses and it greatly reduces the effectiveness of these mechanisms in reality.To address these issues, we propose an innovative anomaly detection strategy called CADM. In CADM, the relative entropy between test logs and normal logs is exploited to discover the anomalous levels. Instead of calculating the relative entropy based on certain predefined data distribution models, our solution inspects the relationship between relative entropy and compression size with an improved grammar-based compression method. No assumptions are needed. In addition, our mechanism has excellent scalability with only O(n) computational complexity. It can generate the detection results on the fly. Experimental analysis with both synthetic and real world logs proves that CADM is superior to the other methods. It can achieve very high anomaly detection accuracy with the minimal computational overhead. It is suitable for log mining tasks and can be applied on a broad variety of application fields.

Log-based software monitoring: a systematic mapping study

“…Researchers have been trying several different techniques, such as deep learning and NLP ( Du et al, 2017 ; Bertero et al, 2017 ; Meng et al, 2019 ; Zhang et al, 2019 ), data mining, statistical learning methods, and machine learning ( Lu et al, 2017 ; He et al, 2016b ; Ghanbari, Hashemi & Amza, 2014 ; Tang & Iyer, 1992 ; Lim, Singh & Yajnik, 2008 ; Xu et al, 2009b , Xu et al, 2009a ) control flow graph mining from execution logs ( Nandi et al, 2016 ), finite state machines ( Fu et al, 2009 ; Debnath et al, 2018 ), frequent itemset mining ( Lim, Singh & Yajnik, 2008 ), dimensionality reduction techniques ( Juvonen, Sipola & Hämäläinen, 2015 ), grammar compression of log sequences ( Gao et al, 2014 ), and probabilistic suffix trees ( Bao et al, 2018 ).…”

PADM: Page Rank-Based Anomaly Detection Method of Log Sequences by Graph Computing

“…We use the method in [7] and the standard HMM for Markov methods, the method in [8] for statistical methods, the T-STDIE [4] method for window based methods and CADM [1] for compression based methods. As the Figure 3 shows, PADM is better than the other methods, demonstrated by the ROC of PADM which is very close to 1.…”

“…Paper [4] and [5] review the existing discrete sequence of anomaly detection methods, and they concluded three different methods of discrete sequence anomaly detection. The anomaly detection work CADM is our previous work [1], which uses the relative entropy between normal logs and testing logs as an identity of anomaly degree. CADM bases on the convergence of the estimation of the divergence, for short sequences, this convergence may not be fully achieved.…”