Continuous whole-system monitoring toward rapid understanding of production HPC applications and systems

Agelastos, Anthony Michael; Allan, Benjamin A.; Brandt, James M.; Gentile, Ann C.; Lefantzi, Sophia; Monk, Steve; Ogden, Jeff; Rajan, Mahesh; Stevenson, Joel O.

doi:10.1016/j.parco.2016.05.009

Cited by 11 publications

(3 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They use machine learning combined with system data, but primarily focus on diagnosing anomalies in compute node health and not performance of jobs. Agelastos et al [10] create a HPC system profiler to explain the performance variability of applications across different HPC systems.…”

Section: Related Workmentioning

confidence: 99%

Analytics of Longitudinal System Monitoring Data for Performance Prediction

Costello,

Bhatele

2020

Preprint

View full text Add to dashboard Cite

In recent years, several HPC facilities have started continuous monitoring of their systems and jobs to collect performance-related data for understanding performance and operational efficiency. Such data can be used to optimize the performance of individual jobs and the overall system by creating data-driven models that can predict the performance of pending jobs. In this paper, we model the performance of representative control jobs using longitudinal system-wide monitoring data to explore the causes of performance variability. Using machine learning, we are able to predict the performance of unseen jobs before they are executed based on the current system state. We analyze these prediction models in great detail to identify the features that are dominant predictors of performance. We demonstrate that such models can be application-agnostic and can be used for predicting performance of applications that are not included in training.

show abstract

Section: Related Workmentioning

confidence: 99%

Analytics of Longitudinal System Monitoring Data for Performance Prediction

Costello,

Bhatele

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, Agelastos et al produced some similar works. In [19], the authors put the focus on the results obtained after doing profiling (at a system and application level) based on global monitoring in an HPC cluster. They collect metrics as we have done, and use a hierarchical model to transmit the data from the compute nodes to aggregating nodes.…”

Section: Related Workmentioning

confidence: 99%

Performance-Aware Scheduling of Parallel Applications on Non-Dedicated Clusters

2019

View full text Add to dashboard Cite

This work presents a HPC framework that provides new strategies for resource management and job scheduling, based on executing different applications in shared compute nodes, maximizing platform utilization. The framework includes a scalable monitoring tool that is able to analyze the platform’s compute node utilization. We also introduce an extension of CLARISSE, a middleware for data-staging coordination and control on large-scale HPC platforms that uses the information provided by the monitor in combination with application-level analysis to detect performance degradation in the running applications. This degradation, caused by the fact that the applications share the compute nodes and may compete for their resources, is avoided by means of dynamic application migration. A description of the architecture, as well as a practical evaluation of the proposal, shows significant performance improvements up to 20% in the makespan and 10% in energy consumption compared to a non-optimized execution.

show abstract

“…Profiling tools that work based on this model provide the most accurate data at high frequencies with a considerably high cost. However, they are more efficient at low frequencies but less informative and prone to detail loss [3]. Current approaches either lack the necessary efficiency to be utilized in production systems or support only post-mortem analysis that does not present online data about application events during the execution.…”

Section: Introductionmentioning

confidence: 99%

Production Application Performance Data Streaming for System Monitoring

Izadpanah

Allan

Dechev

et al. 2019

ACM Trans. Model. Perform. Eval. Comput. Syst.

Self Cite

View full text Add to dashboard Cite

In this article, we present an approach to streaming collection of application performance data. Practical application performance tuning and troubleshooting in production high-performance computing (HPC) environments requires an understanding of how applications interact with the platform, including (but not limited to) parallel programming libraries such as Message Passing Interface (MPI). Several profiling and tracing tools exist that collect heavy runtime data traces either in memory (released only at application exit) or on a file system (imposing an I/O load that may interfere with the performance being measured). Although these approaches are beneficial in development stages and post-run analysis, a systemwide and low-overhead method is required to monitor deployed applications continuously. This method must be able to collect information at both the application and system levels to yield a complete performance picture. In our approach, an application profiler collects application event counters. A sampler uses an efficient inter-process communication method to periodically extract the application counters and stream them into an infrastructure for performance data collection. We implement a tool-set based on our approach and integrate it with the Lightweight Distributed Metric Service (LDMS) system, a monitoring system used on large-scale computational platforms. LDMS provides the infrastructure to create and gather streams of performance data in a low overhead manner. We demonstrate our approach using applications implemented with MPI, as it is one of the most common standards for the development of large-scale scientific applications. We utilize our tool-set to study the impact of our approach on an open source HPC application, Nalu. Our tool-set enables us to efficiently identify patterns in the behavior of the application without source-level knowledge. We leverage LDMS to collect system-level performance data and explore the correlation between the system and application events. Also, we demonstrate how our tool-set can help detect anomalies with a low latency. We run tests on two different architectures: a system enabled with Intel Xeon Phi and another system equipped with Intel Xeon processor. Our overhead study shows our method imposes at most 0.5% CPU usage overhead on the application in realistic deployment scenarios.

show abstract

Continuous whole-system monitoring toward rapid understanding of production HPC applications and systems

Cited by 11 publications

References 8 publications

Analytics of Longitudinal System Monitoring Data for Performance Prediction

Analytics of Longitudinal System Monitoring Data for Performance Prediction

Performance-Aware Scheduling of Parallel Applications on Non-Dedicated Clusters

Production Application Performance Data Streaming for System Monitoring

Contact Info

Product

Resources

About