Machine Learning-based Anomaly Detection of Ganglia Monitoring Data in HEP Data Center

Abstract

This paper introduces a generic and scalable anomaly detection framework. Anomaly detection can improve operation and maintenance efficiency and assure experiments can be carried out effectively. The framework facilitates common tasks such as data sample building, retagging and visualization, deviation measurement and performance measurement for machine learning-based anomaly detection methods. The samples we used are sourced from Ganglia monitoring data. There are several anomaly detection methods to handle spatial and temporal anomalies within the framework. Finally, we show the rudimental application of the framework on Lustre distributed file systems in daily operation and maintenance.

Operation and maintenance analysis platform at IHEP

“…classification algorithms, such as isolated forest algorithm, and clustering algorithms: such as K-means, etc. [11] Feature matching is performed on the preprocessed operation and maintenance data, and anomalies are automatically classified. Or according to the judgment rules provided by the expert experience database, key monitoring metrics are used for threshold judgment, historical data year-on-year, chainmonth correlation calculation, and other processing methods to detect an anomaly.…”