Robust log-based anomaly detection on unstable log data

Anomaly Detection via Mining Numerical Workflow Relations from Logs

“…It designs a fast cascading clustering algorithm for iteratively sampling, clustering, and matching log sequences and then identifies the problems by correlating the clusters with system KPIs. To identify and handle unstable log events and sequences, LogRobust [14] was proposed, which can extract semantic information of log events and then detects anomalies by utilizing an attention-based Bi-LSTM model. Though He et al [7] has found that supervised approaches usually achieve better performances than unsupervised approaches, the biggest concern for the supervised approaches is that they require labelled logs to train the models.…”

“…sessions, fixed windows, and sliding windows. Grouping by sessions is to classify the log events by certain identifiers, such as TaskID, InstanceID, or BlockID [14], [15], [7]. Grouping by fixed or sliding windows are based on the timestamps of the log entries.…”

“…where c mn indicates the number of occurrences of Event n in session m. Over the years, many machine learning based methods have been proposed to employ the event-count-matrix to detect system anomalies [14], [16], [17], [18], [19], [20], [21], [22], [23]. Some of these methods use supervised learning techniques such as Logistic Regression [1], Decision Trees [2], and Support Vector Machines (SVMs) [3], while others employ unsupervised approaches such as Log Clustering [4], Principal Component Analysis [5], and the previously mentioned Invariant Miner (IM) [6].…”

Priolog: Mining Important Logs via Temporal Analysis and Prioritization

“…There has been extensive research in the detection of anomalies or outliers in logs using both machine learning approaches and using relations across multivariate time-series data in several application domains [14,17,18,[24][25][26][27][28][29][30][31]. In this section, we review a set of representative examples of outlier detection applied to log analysis, and highlight a key focus of the contributions of our paper in the context of these rich body of prior art.…”

“…The method does not leverage domain knowledge or filter for false positives as done in our current work. The most recent work in the log anomaly detection field is the LogRobust [31]. Zhang et al have developed a Bi-LSTM classification model from the fixed dimension semantic vectors of logs and improved the anomaly detection capability.…”