An Explainable Model for Fault Detection in HPC Systems

Abstract

Large supercomputers are composed of numerous components that risk to break down or behave in unwanted manners. Identifying broken components is a daunting task for system administrators. Hence an automated tool would be a boon for the systems resiliency. The wealth of data available in a supercomputer can be used for this task. In this work we propose an approach to take advantage of holistic data centre monitoring, system administrator node status labeling and an explainable model for fault detection in supercomputing nodes. The proposed model aims at classifying the different states of the computing nodes thanks to the labeled data describing the supercomputer behaviour, data which is typically collected by system administrators but not integrated in holistic monitoring infrastructure for data center automation. In comparison the other method, the one proposed here is robust and provide explainable predictions. The model has been trained and validated on data gathered from a tier-0 supercomputer in production.

Ruad: Unsupervised Anomaly Detection in Hpc Systems

“…The proposed model architecture achieves the highest AUC of 0.77 compared to 0.75, which is the highest AUC achieved by the dense autoencoders (on our dataset). Another contribution of this paper is that the proposed method -unlike the previous work [5,16,17,8] achieves the best results in an unsupervised training case. Unsupervised training is instrumental as it offers a possibility of deploying an anomaly detection model to the cases where (accurately) labelled dataset is unavailable.…”

“…All mentioned approaches use synthetic anomalies injected into the HPC system to train a supervised classification model. Approaches [5] and [16] are among the few that leverage real anomalies collected from production HPC systems (as opposed to injected anomalies). In this paper, we are interested in real anomalies, and thus, we will not include methods using synthetic/simulated data or injected anomalies in our quantitative comparisons.…”

“…So far, the challenges of anomaly detection on HPC systems have been approached by deploying anomaly reporting tools by training the models in a supervised or semi-supervised fashion [5,16,17,8]. The need for an accurately labelled training set is the main limitation of current approaches as it is expensive, in terms of time and effort of the system administrators, to be applied in practice.…”

Application of multivariate time-series model for high performance computing (HPC) fault prediction

“…In recent years, owing to the increasing demand for high-performance computing (HPC) as well as the scale-up supercomputers and intelligent computing systems, the reliability of largescale computing systems has been investigated extensively [1][2][3][4]. The system operation is complex, and failures occur frequently which are difficult to detect, locate, diagnose, analyze, and debug [1,5,6].…”