Accurate fault prediction of BlueGene/P RAS logs via geometric reduction

Abstract

This investigation presents two distinct and novel approaches for the prediction of system failures occurring inOak Ridge National Laboratory's Blue Gene/P supercomputer. Each technique uses raw numeric and textual subsets of large data logs of physical system information such as fan speeds and CPU temperatures. This data is used to develop models of the system capable of sensing anomalies, or deviations from nominal behavior. Each algorithm predicted event log reported anomalies in advance of their occurrence and one algorithm did so without false positives. Both algorithms predicted an anomaly that did not appear in the event log. It was later learned that the fault missing from the log but predicted by both algorithms was confirmed to have occurred by the system administrator.

Filtering log data: Finding the needles in the Haystack

“…Our work is inspired by the Huan's work [29] and Thompson's work [30]. In [29], a KD-tree based sampling method was proposed for informative instance selection.…”

“…In this paper, we modify the KD-tree algorithm to rule out instances irrelevant to abnormal states. In [30], Thompson utilized two feature extraction methods on numeric data for failure prediction in the Blue Gene/P system, which was based on the same dataset in our first case study. However, their study was only on the feature level, which was insufficient for data reduction.…”

A GPU-Oriented Algorithm Design for Secant-Based Dimensionality Reduction

“…After initializing P (1) , the algorithm proceeds as follows: at each iteration i we calculate the 2 -norms of all secants under the current projection. In other words we find the column of P (i) T S with the smallest 2 norm; call this column index j * .…”

“…Figure 2 shows the projection into R 3 obtained from PCA (i.e. the projection of the data from R 10 to R 3 via Pr (1) ), and Figure 3 shows the projection onto R 3 obtained from 100 iterations of the Secant-Avoidance Projection algorithm. In each case, Fig.…”

“…Even the question of monitoring these high-performance computing systems is a challenge. Mathematical models for anomaly detection provide an interesting direction, but these approaches also suffer from the curse of dimensionality [1], [2]. Characterizing the behavior of a 100,000,000 variable dynamical system requires new modeling strategies.…”