Large-Scale Persistent Numerical Data Source Monitoring System Experiences

Brandt, James M.; Gentile, Ann C.; Showerman, Michael; Enos, Jeremy; Fullop, Joshi; Bauer, Gregory H.

doi:10.1109/ipdpsw.2016.188

Cited by 5 publications

(4 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Accounts of long-term production experiences, however, are currently still rare and mostly cover ODAV deployments. Brandt et al [10] discuss their 2-year LDMS deployment on Blue Waters, a large-scale HPC system with more than 27,000 nodes. The authors describe a series of technical challenges related to reliability, overhead and data consistency, as well as dive into system-specific issues, such as clock skew effects.…”

Section: State Of the Artmentioning

confidence: 99%

Operational Data Analytics in Practice: Experiences from Design to Deployment in Production HPC Environments

Netti¹,

Ott²,

Guillen³

et al. 2021

Preprint

View full text Add to dashboard Cite

As HPC systems grow in complexity, efficient and manageable operation is increasingly critical. Many centers are thus starting to explore the use of Operational Data Analytics (ODA) techniques, which extract knowledge from massive amounts of monitoring data and use it for control and visualization purposes. As ODA is a multifaceted problem, much effort has gone into researching its separate aspects: however, accounts of production ODA experiences are still hard to come across.In this work we aim to bridge the gap between ODA research and production use by presenting our experiences with ODA in production, involving in particular the control of cooling infrastructures and visualization of job data on two HPC systems. We cover the entire development process, from design to deployment, highlighting our insights in an effort to drive the community forward. We rely on open-source tools, which make for a generic ODA framework suitable for most scenarios.

show abstract

Section: State Of the Artmentioning

confidence: 99%

Operational Data Analytics in Practice: Experiences from Design to Deployment in Production HPC Environments

Netti¹,

Ott²,

Guillen³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Establishing the necessary framework for holistic and continuous monitoring of large-scale HPC systems and their infrastructure is extremely challenging in many ways [4,9].…”

Section: Monitoring Challengesmentioning

confidence: 99%

From Facility to Application Sensor Data: Modular, Continuous and Holistic Monitoring with DCDB

Netti,

Mueller,

Auweter

et al. 2019

Preprint

View full text Add to dashboard Cite

Today's HPC installations are highly-complex systems, and their complexity will only increase as we move to exascale and beyond. At each layer, from facilities to systems, from runtimes to applications, a wide range of tuning decisions must be made in order to achieve efficient operation. This, however, requires systematic and continuous monitoring of system and user data. While many insular solutions exist, a system for holistic and facility-wide monitoring is still lacking in the current HPC ecosystem.In this paper we introduce DCDB, a comprehensive monitoring system capable of integrating data from all system levels. It is designed as a modular and highly-scalable framework based on a plugin infrastructure. All monitored data is aggregated at a distributed noSQL data store for analysis and cross-system correlation. We demonstrate the performance and scalability of DCDB, and describe two use cases in the area of energy management and characterization.

show abstract

“…Current approaches either lack the necessary efficiency to be utilized in production systems or support only post-mortem analysis that does not present online data about application events during the execution. Also, developers face challenges when analyzing applications that scale to larger parallel systems [9].…”

Section: Introductionmentioning

confidence: 99%

“…We integrate our approach with the Lightweight Distributed Metric Service (LDMS) system [2], a monitoring system used on large-scale computational platforms [9]. LDMS provides the infrastructure to gather streams of performance data efficiently while keeping the overhead low.…”

Section: Introductionmentioning

confidence: 99%

Production Application Performance Data Streaming for System Monitoring

Izadpanah

Allan

Dechev

et al. 2019

ACM Trans. Model. Perform. Eval. Comput. Syst.

View full text Add to dashboard Cite

In this article, we present an approach to streaming collection of application performance data. Practical application performance tuning and troubleshooting in production high-performance computing (HPC) environments requires an understanding of how applications interact with the platform, including (but not limited to) parallel programming libraries such as Message Passing Interface (MPI). Several profiling and tracing tools exist that collect heavy runtime data traces either in memory (released only at application exit) or on a file system (imposing an I/O load that may interfere with the performance being measured). Although these approaches are beneficial in development stages and post-run analysis, a systemwide and low-overhead method is required to monitor deployed applications continuously. This method must be able to collect information at both the application and system levels to yield a complete performance picture. In our approach, an application profiler collects application event counters. A sampler uses an efficient inter-process communication method to periodically extract the application counters and stream them into an infrastructure for performance data collection. We implement a tool-set based on our approach and integrate it with the Lightweight Distributed Metric Service (LDMS) system, a monitoring system used on large-scale computational platforms. LDMS provides the infrastructure to create and gather streams of performance data in a low overhead manner. We demonstrate our approach using applications implemented with MPI, as it is one of the most common standards for the development of large-scale scientific applications. We utilize our tool-set to study the impact of our approach on an open source HPC application, Nalu. Our tool-set enables us to efficiently identify patterns in the behavior of the application without source-level knowledge. We leverage LDMS to collect system-level performance data and explore the correlation between the system and application events. Also, we demonstrate how our tool-set can help detect anomalies with a low latency. We run tests on two different architectures: a system enabled with Intel Xeon Phi and another system equipped with Intel Xeon processor. Our overhead study shows our method imposes at most 0.5% CPU usage overhead on the application in realistic deployment scenarios.

show abstract

Large-Scale Persistent Numerical Data Source Monitoring System Experiences

Cited by 5 publications

References 4 publications

Operational Data Analytics in Practice: Experiences from Design to Deployment in Production HPC Environments

Operational Data Analytics in Practice: Experiences from Design to Deployment in Production HPC Environments

From Facility to Application Sensor Data: Modular, Continuous and Holistic Monitoring with DCDB

Production Application Performance Data Streaming for System Monitoring

Contact Info

Product

Resources

About