Comprehensive Resource Use Monitoring for HPC Systems with TACC Stats

Evans, R. T.; Barth, William L.; Browne, James C.; DeLeon, Robert L.; Furlani, Thomas R.; Gallo, Steven M.; Jones, Matthew D.; Patra, Abani

doi:10.1109/hust.2014.7

Cited by 56 publications

(24 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The TACC Stats [19] project also shares many of our goals. TACC Stats is used by admins for understanding, but the data is aggregated nightly and hence cannot be used for run-time and immediate post-run understanding.…”

Section: Related Workmentioning

confidence: 98%

Continuous whole-system monitoring toward rapid understanding of production HPC applications and systems

et al. 2016

View full text Add to dashboard Cite

Highlights• Monitoring can provide meaningful system and application profiling in production.• Visual and analytical characterizations can inform usage and procurement decisions.• Resource utilization scoring provides simple but informative characterizations.• Continuous, synchronous, high-fidelity, whole-system monitoring is required. AbstractA detailed understanding of HPC applications' resource needs and their complex interactions with each other and HPC platform resources is critical to achieving scalability and performance. Such understanding has been difficult to achieve because typical application profiling tools do not capture the behaviors of codes under the potentially wide spectrum of actual production conditions and because typical monitoring tools do not capture system resource usage information with high enough fidelity to gain sufficient insight into application performance and demands.In this paper we present both system and application profiling results based on data obtained through synchronized system wide monitoring on a production HPC cluster at Sandia National Laboratories (SNL). We demonstrate analytic and visualization techniques that we are using to characterize application and system resource usage under production conditions for better understanding of application resource needs. Our goals are to improve application performance (through understanding application-to-resource mapping and system throughput) and to ensure that future system capabilities match their intended workloads.

show abstract

Section: Related Workmentioning

confidence: 98%

Continuous whole-system monitoring toward rapid understanding of production HPC applications and systems

et al. 2016

View full text Add to dashboard Cite

show abstract

“…This cluster consists of 1888 computing nodes from which the records of 1709 nodes are used in this study. The resource usage data is collected using the TACC Stats system monitor [10] which records various resources usage statistics at each computational node every 10 minutes. In our experiments, we use a set of 86 resource usage statistics with a resolution of 10 minutes from 1:10:01 March 1 st 2013 to 23:40:01 March 7 th 2013.…”

Section: Datasetmentioning

confidence: 99%

“…Several tools exist for collecting and visualizing resource usage data from large scale HPC installations (e.g. Texas Advanced Computing Center TACC Stats [10], XSEDE Metrics on Demand or XDMoD [17], etc.). Such tools can produce large amounts of high dimensional resource usage data at a high temporal frequency for each computational node in the system.…”

Section: Introductionmentioning

confidence: 99%

dynamicMF: A Matrix Factorization Approach to Monitor Resource Usage in High Performance Computing Systems

Sorkunlu

Luong

Chandola

2018

2018 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

High performance computing (HPC) facilities consist of a large number of interconnected computing units (or nodes) that execute highly complex scientific simulations to support scientific research. Monitoring such facilities, in real-time, is essential to ensure that the system operates at peak efficiency. Such systems are typically monitored using a variety of measurement and log data which capture the state of the various components within the system at regular intervals of time. As modern HPC systems grow in capacity and complexity, the data produced by current resource monitoring tools is at a scale that it is no longer feasible to be visually monitored by analysts. We propose a method that transforms the multi-dimensional output of resource monitoring tools to a low dimensional representation that facilitates the understanding of the behavior of a High Performance Computing (HPC) system. The proposed method automatically extracts the low-dimensional signal in the data which can be used to track the system ef-ficiency and identify performance anomalies. The method models the resource usage data as a three dimensional tensor (capturing resource usage of all compute nodes for difference resources over time). A dynamic matrix factorization algorithm, called dynamicMF, is proposed to extract a low-dimensional temporal signal for each node, which is subsequently fed into an anomaly detector. Results on resource usage data collected from the Lonestar 4 system at the Texas Advanced Computing Center show that the identified anomalies are correlated with actual anomalous events reported in the system log messages.

show abstract

“…Many of these solutions are site-or vendor-specific and are thus not easy to deploy at other sites. A solution that also targets small-to medium-sized clusters is TACC Stats [6], which is also used as part of the larger XDMoD project [7]. Recent and current efforts include the FEPA project [8], from which also the approach presented in this paper originates, and the just-started ProfiT-HPC [9].…”

Section: Related Workmentioning

confidence: 99%

LIKWID Monitoring Stack: A Flexible Framework Enabling Job Specific Performance monitoring for the masses

Röhl

Eitzinger

Hager

et al. 2017

2017 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

Abstract-System monitoring is an established tool to measure the utilization and health of HPC systems. Usually system monitoring infrastructures make no connection to job information and do not utilize hardware performance monitoring (HPM) data. To increase the efficient use of HPC systems automatic and continuous performance monitoring of jobs is an essential component. It can help to identify pathological cases, provides instant performance feedback to the users, offers initial data to judge on the optimization potential of applications and helps to build a statistical foundation about application specific system usage. The LIKWID monitoring stack is a modular framework build on top of the LIKWID tools library. It aims on enabling job specific performance monitoring using HPM data, system metrics and applicationlevel data for small to medium sized commodity clusters. Moreover, it is designed to integrate in existing monitoring infrastructures to speed up the change from pure system monitoring to job-aware monitoring.

show abstract

Comprehensive Resource Use Monitoring for HPC Systems with TACC Stats

Cited by 56 publications

References 3 publications

Continuous whole-system monitoring toward rapid understanding of production HPC applications and systems

Continuous whole-system monitoring toward rapid understanding of production HPC applications and systems

dynamicMF: A Matrix Factorization Approach to Monitor Resource Usage in High Performance Computing Systems

LIKWID Monitoring Stack: A Flexible Framework Enabling Job Specific Performance monitoring for the masses

Contact Info

Product

Resources

About