Detecting large-scale system problems by mining console logs

Abstract

Surprisingly, console logs rarely help operators detect problems in large-scale datacenter services, for they often consist of the voluminous intermixing of messages from many software components written by independent developers. We propose a general methodology to mine this rich source of information to automatically detect system runtime problems. We first parse console logs by combining source code analysis with information retrieval to create composite features. We then analyze these features using machine learning to detect operational problems. We show that our method enables analyses that are impossible with previous methods because of its superior ability to create sophisticated features. We also show how to distill the results of our analysis to an operator-friendly one-page decision tree showing the critical messages associated with the detected problems. We validate our approach using the Darkstar online game server and the Hadoop File System, where we detect numerous real problems with high accuracy and few false positives. In the Hadoop case, we are able to analyze 24 million lines of console logs in 3 minutes. Our methodology works on textual console logs of any size and requires no changes to the service software, no human input, and no knowledge of the software's internals.

Performance Analysis Tool for HPC and Big Data Applications on Scientific Clusters

“…Barham et al [5] proposed to use clustering to identify anomalous requests. Xu et al [38] proposed to find erroneous execution paths using the PCA [19] on console logs. Bod et al [6] used logistic regression with L1 regularization on the vector of metric quantiles to fingerprint performance crisis.…”

A Lightweight Tool for Anomaly Detection in Cloud Data Centres

“…Moreover, modern systems consisting of multiple components developed by different vendors and the frequent upgrades of those components make it difficult for a single expert to have complete knowledge of the total system and to set the rules effectively. This complexity has given rise to statistical learning based log analytic tools such as the works of Lou et al (Lou et al, 2010) and Xu et al (Xu et al, 2009), which extract features from console logs and then use statistical techniques to automatically build models for system anomaly identification. Lou et al (Lou et al, 2010) propose a statistical learning technique which consists of a learning process and a detection process.…”

“…Each mismatch in the invariants is considered to be anomalous. Xu et al (Xu et al, 2009) propose a new methodology to mine console logs to automatically detect system problems. This first creates feature vectors from the logs and then applies the PCA (Principal Component Analysis) algorithm on the feature vectors to detect anomalies.…”

“…Log analytic tools (Lou et al, 2010;Xu et al, 2009) extract features from logs and use statistical learning techniques to automatically build models that detect and diagnose system anomalies in data centres. These models are not easy to understand by human operators as they may detect problems in a high dimensional feature space without providing meaningful explanations for the detected problems.…”

META: Multi-resolution Framework for Event Summarization

“…Different analysis tasks pay attention to different application aspects, such as system failure tracing [19,10], event correlation discovery [20,18,16], and event based trend analysis [5,6,7]. In practice, these methods are often conducted when the analysts already have some prior knowledge about the data.…”