Correlation-wise Smoothing: Lightweight Knowledge Extraction for HPC Monitoring Data

Abstract

Modern High-Performance Computing (HPC) and data center operators rely more and more on data analytics techniques to improve the efficiency and reliability of their operations. They employ models that ingest time-series monitoring sensor data and transform it into actionable knowledge for system tuning: a process known as Operational Data Analytics (ODA). However, monitoring data has a high dimensionality, is hardware-dependent and difficult to interpret. This, coupled with the strict requirements of ODA, makes most traditional data mining methods impractical and in turn renders this type of data cumbersome to process. Most current ODA solutions use ad-hoc processing methods that are not generic, are sensible to the sensors' features and are not fit for visualization.In this paper we propose a novel method, called Correlationwise Smoothing (CS), to extract descriptive signatures from timeseries monitoring data in a generic and lightweight way. Our CS method exploits correlations between data dimensions to form groups and produces image-like signatures that can be easily manipulated, visualized and compared. We evaluate the CS method on HPC-ODA, a collection of datasets that we release with this work, and show that it leads to the same performance as most state-of-the-art methods while producing signatures that are up to ten times smaller and up to ten times faster, while gaining visualizability, portability across systems and clear scaling properties.

Operational Data Analytics in Practice: Experiences from Design to Deployment in Production HPC Environments

“…Further, the models are naturally biased since they are trained to predict the maximum temperature rather than the average, leading to a natural tendency to over-estimation: this is a deliberate choice in order to make the control pipeline more robust against over-heating. Finally, using the CS method gives us compact models that have a negligible impact on overhead and that are resistant against changes in the available set of sensors over time: after proving its validity through tailored experiments in previous work, this is the first real-life use case for this technique [38]. 4.5.2 Impact on Infrastructure.…”

“…Based on an extensive survey of experimental techniques proposed in the literature [3, 13, 18, 20, 21, 23, 26, 30, 34, 35, 40-42, 45, 47] and in light of our long-term experiences at LRZ [36][37][38], we derive a generic formulation for ODA -namely, we identify the main functional steps composing it. These are summarized in Figure 1: the first step consists in Monitoring of system resources by collecting sensor or log data; this is followed by Monitoring Data Processing, transforming the raw monitoring data into a polished representation that can be comprehended by ODA techniques -aggregation and dimensionality reduction, for example, are two common approaches to solve this task.…”

“…The single steps of ODA entail different operational requirements for working effectively in production environments. As these have been explored separately in previous work [36][37][38], here we focus on the overarching requirements of an ODA pipeline as a whole:…”

“…Moreover, Wintermute supplies a series of abstraction constructs (denoted as block system) that allow to write compact configurations, even for complex ODA models deployed at a large scale. • Correlation-wise Smoothing (CS) [38] is a monitoring data processing technique designed to compress sensor data into compact signatures that are descriptive of a component's status. The signatures are built by arranging data dimensions on a bi-dimensional space, where the horizontal axis represents time and the vertical one the data dimensions, with the latter ordered according to their correlations.…”

“…Then, CM and ESB Pushers implement a pipeline to predict CPU and GPU temperatures, used to steer control decisions for the associated RCU. This is supported by two operator plugins: first, a CS operator plugin provides an online implementation of the homonymous technique [38], which converts the data associated with a set of input sensors (both raw and produced by the Aggregator plugin) into a compact signature made of 20 blocks (i.e., complex coefficients). The resulting 40 sensors are then leveraged by a Regressor plugin, which implements random forest-based regression using the OpenCV library 7 -here, an ML model is configured to use as feature vectors the CS signatures and produce as output a prediction of the maximum expected CPU or GPU temperature in the near future.…”