Global Experiences with HPC Operational Data Measurement, Collection and Analysis

Ott, Michael; Shin, Woong; Bourassa, Norman; Wilde, Torsten; Ceballos, Stefan; Romanus, Melissa; Bates, Natalie

doi:10.1109/cluster49012.2020.00071

Cited by 19 publications

(11 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To answer these research questions, in this manuscript we extend Examon, a state-of-the-art ODA framework [13], [15], deployed on the CINECA 1 data center, to integrate Nagios [16] monitored events. We are the first in exploring the possibility to use Nagios as an annotation tool (to provide normal and faulty state labels).…”

Section: A Contributionsmentioning

confidence: 99%

“…As the size of supercomputing systems approaches the exascale, it is common to adopt Operational Data measurement, collection and Analysis (ODA) frameworks [13] to continuously monitor system information data (mostly in the form of multivariate time series data), such as data coming from physical sensors' telemetry (temperature, power), micro-architectural events (IPC, cache misses), data coming from the computing resources and facility [13]- [15]. These do not contain the records of node's and system failure events.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Anomaly Detection and Anticipation in High Performance Computing Systems

Borghesi

Molan

Milano

et al. 2022

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

In their quest towards Exascale, High Performance Computing (HPC) systems are rapidly becoming larger and more complex, together with the issues concerning their maintenance. Luckily, many current HPC systems are endowed with data monitoring infrastructures that characterize the system state, and whose data can be used to train Deep Learning (DL) anomaly detection models, a very popular research area. However, the lack of labels describing the state of the system is a wide-spread issue, as annotating data is a costly task, generally falling on human system administrators and thus does not scale toward exascale.In this work we investigate the possibility to extract labels from a service monitoring tool (Nagios) currently used by HPC system administrators to flag the nodes which undergo maintenance operations. This allows to automatically annotate data collected by a fine-grained monitoring infrastructure; this labelled data is then used to train and validate a DL model for anomaly detection. We conduct the experimental evaluation on a tier-0 production supercomputer hosted at CINECA, Bologna, Italy. The results reveal that the DL model can accurately detect the real failures, and, moreover, it can predict the insurgency of anomalies, by systematically anticipating the actual labels (i.e. the moment when system administrators realize when an anomalous event happened); the average advance time computed on historical traces is around 45 minutes. The proposed technology can be easily scaled toward exascale systems to easy their maintenance.

show abstract

Section: A Contributionsmentioning

confidence: 99%

mentioning

confidence: 99%

Anomaly Detection and Anticipation in High Performance Computing Systems

Borghesi

Molan

Milano

et al. 2022

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

show abstract

“…This often leads to multiple tools being used, in turn resulting in complex and fragmented software stacks and in a wide disarray of monitoring data that is difficult to use effectively [22]. This statement is confirmed in a survey conducted by the Energy Efficient HPC Working Group (EEHPCWG) in 2019 [39] regarding the use of ODA in several HPC centers: most sites employ varying sets of monitoring, storage and analysis solutions that either rely on in-house systems, or on commercial products that are not tailored for data center monitoring, thus restricting administrators to simple ODAV visual inspection.…”

Section: State Of the Artmentioning

confidence: 99%

“…Each of these individual steps has been explored thoroughly in the ODA research field, but there is currently a severe lack of end-to-end experiences, from design down to maintenance, covering all aspects of the ODA pipeline and providing the necessary insights and solutions to propel forward the capillary adoption of ODA in production data center environments. It has been observed in the literature, in fact, that most HPC centers rely on insular ODA solutions tackling only specific aspects of the problem [39], with no clear applicability to other domains.…”

Section: Introductionmentioning

confidence: 99%

Operational Data Analytics in Practice: Experiences from Design to Deployment in Production HPC Environments

Netti¹,

Ott²,

Guillen³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

As HPC systems grow in complexity, efficient and manageable operation is increasingly critical. Many centers are thus starting to explore the use of Operational Data Analytics (ODA) techniques, which extract knowledge from massive amounts of monitoring data and use it for control and visualization purposes. As ODA is a multifaceted problem, much effort has gone into researching its separate aspects: however, accounts of production ODA experiences are still hard to come across.In this work we aim to bridge the gap between ODA research and production use by presenting our experiences with ODA in production, involving in particular the control of cooling infrastructures and visualization of job data on two HPC systems. We cover the entire development process, from design to deployment, highlighting our insights in an effort to drive the community forward. We rely on open-source tools, which make for a generic ODA framework suitable for most scenarios.

show abstract

“…If we keep track of application executions and recognize that a job executes a known application, we can: (a) make predictions about resource usage based on executions in the past (improving job scheduling [14] and predicting energy consumption [12]), (b) detect deviations from past resource usage (indicating anomalies and potential errors), (c) detect resource usage of known malicious applications (e.g. cryptocurrency mining [5]), and (d) lower power consumption by reducing CPU frequency for memorybound applications [10].…”

Section: Introductionmentioning

confidence: 99%

An Execution Fingerprint Dictionary for HPC Application Recognition

Thomas¹,

Lachiche²,

Cavelan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Applications running on HPC systems waste time and energy if they: (a) use resources inefficiently, (b) deviate from allocation purpose (e.g. cryptocurrency mining), or (c) encounter errors and failures. It is important to know which applications are running on the system, how they use the system, and whether they have been executed before. To recognize known applications during execution on a noisy system, we draw inspiration from the way Shazam recognizes known songs playing in a crowded bar. Our contribution is an Execution Fingerprint Dictionary (EFD) that stores execution fingerprints of system metrics (keys) linked to application and input size information (values) as key-value pairs for application recognition. Related work often relies on extensive system monitoring (many system metrics collected over large time windows) and employs machine learning methods to identify applications. Our solution only uses the first 2 minutes and a single system metric to achieve F-scores above 95 percent, providing comparable results to related work but with a fraction of the necessary data and a straightforward mechanism of recognition.

show abstract

Global Experiences with HPC Operational Data Measurement, Collection and Analysis

Cited by 19 publications

References 16 publications

Anomaly Detection and Anticipation in High Performance Computing Systems

Anomaly Detection and Anticipation in High Performance Computing Systems

Operational Data Analytics in Practice: Experiences from Design to Deployment in Production HPC Environments

An Execution Fingerprint Dictionary for HPC Application Recognition

Contact Info

Product

Resources

About