Data Stream Clustering for Online Anomaly Detection in Cloud Applications

Abstract

This paper introduces a new approach for the online detection of performance anomalies in cloud virtual machines (VMs). It is designed for cloud infrastructure providers to detect during runtime unknown anomalies that may still be observed in complex modern systems hosted on VMs. The approach is drawn on data stream clustering of per-VM monitoring data and detects at a fine granularity where anomalies occur. It is design to be independent of the types of applications deployed over VMs. Moreover it deals with relentless changes in systems normal behaviors during runtime. The parallel analyses of each VM makes this approach scalable to a large number of VMs composing an application. The approach consists of two online steps: 1) the incremental update of sets of clusters by means of data stream clustering, and 2) the computation of two attributes characterizing the global clusters evolution. We validate our approach over a VMware vSphere testbed. It hosts a typical cloud application, MongoDB, that we study in normal behavior contexts and in presence of anomalies.

Anomaly Detection and Root Cause Localization in Virtual Network Functions

“…Besides, this work particularly focuses on the identification of anomalous behaviors from monitoring data of VMs OSs such as CPU consumption, disk I/O, and free memory. As seen in previous researches [10]- [16], these data are well suited to reflect the behavior of a computing system. With respect to the machine learning models that we aim to build for detection (in our case, classifiers), samples of labeled monitoring data are needed to train them to discern different VNF behaviors.…”

Anomaly detection and diagnosis for cloud services: Practical experiments and lessons learned

“…In that scenario, the models from supervised learning would detect the most common anomalies. As for the outputs of the models from unsupervised learning (e.g., based on the clustering of observations like in our previous work Sauvanaud et al (2015a)), they would be considered on periods during which the models from supervised learning would not detect any anomaly, and possibly missing some unknown ones.…”

“…Unsupervised learning does not require such knowledge and enables the detection of unknown anomalies. While our ADS can work with both algorithms, as shown in our previous work Sauvanaud et al (2015a), in this paper we focus on supervised algorithms. Since a cloud service is mostly exhibiting a normal behavior during runtime, we developed fault injection tools for two goals: i) to inject anomalies during the training phase of machine learning models and collect data of both normal behaviors and anomalies, ii) to assess the anomaly detection and diagnosis efficiency of our ADS during a validation phase in presence of anomalies.…”

Anomaly detecting and ranking of the cloud computing platform by multi-view learning

“…Sauvanaud, etc. exploit the dynamic index set both to update anomaly class and to compute the classification of the mean value adapting the changing of data distribution [18]. Fu, etc.…”