Designing a Profiling and Visualization Tool for Scalable and In-depth Analysis of High-Performance GPU Clusters

Kousha, Pouya; Ramesh, Bharath; Suresh, Kaushik Kandadi; Chu, C. Y. Cyrus; Jain, Arpan; Sarkauskas, Nick; Subramoni, Hari; Panda, Dhabaleswar K.

doi:10.1109/hipc.2019.00022

Cited by 14 publications

(4 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, there are tools built on top of existing NVIDIA libraries. For example, nvtop [6] is a wrapper around NVML that provides visualization for NVML metrics, and Moneo [24] is a monitoring system that specifically targets AI applications.…”

Section: Related Workmentioning

confidence: 99%

“…Orthogonal to Nsight Systems and Nsight Compute, there has also been efforts to build profiling tools using the NVIDIA CUDA Profiling Tools Interface (CUPTI API [26]) [12,20,24,31,38,39]. Furthermore, there are tools that have a stronger focus on profiling the data movement, such as [20], which is also built on top of CUPTI in addition to OSU INAM [33].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Profiling and Monitoring Deep Learning Training Tasks

Yousefzadeh-Asl-Miandoab

Robroek

Tözün

2023

Proceedings of the 3rd Workshop on Machine Learning and Systems

View full text Add to dashboard Cite

The embarrassingly parallel nature of deep learning training tasks makes CPU-GPU co-processors the primary commodity hardware for them. The computing and memory requirements of these tasks, however, do not always align well with the available GPU resources. It is, therefore, important to monitor and profile the behavior of training tasks on co-processors to understand better the requirements of different use cases. In this paper, our goal is to shed more light on the variety of tools for profiling and monitoring deep learning training tasks on server-grade NVIDIA GPUs. In addition to surveying the main characteristics of the tools, we analyze the functional limitations and overheads of each tool by using a both light and heavy training scenario. Our results show that monitoring tools like nvidia-smi and dcgm can be integrated with resource managers for online decision making thanks to their low overheads. On the other hand, one has to be careful about the set of metrics to correctly reason about the GPU utilization. When it comes to profiling, each tool has its time to shine; a framework-based or systemwide GPU profiler can first detect the frequent kernels or bottlenecks, and then, a lower-level GPU profiler can focus on particular kernels at the micro-architectural-level.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Profiling and Monitoring Deep Learning Training Tasks

Yousefzadeh-Asl-Miandoab

Robroek

Tözün

2023

Proceedings of the 3rd Workshop on Machine Learning and Systems

View full text Add to dashboard Cite

show abstract

“…Welton and Miller [22] investigated hidden performance issues that impact several HPC applications but are not reported by tool APIs. Kousha et al [23] developed a tool for monitoring communications on multiple GPUs. Unlike the aforementioned tools, HPCToolkit collects call path profiles and shows calling context information in both trace and profile views.…”

Section: Related Workmentioning

confidence: 99%

Measurement and analysis of GPU-accelerated applications with HPCToolkit

et al. 2021

View full text Add to dashboard Cite

To address the challenge of performance analysis on the US DOE's forthcoming exascale supercomputers, Rice University has been extending its HPCToolkit performance tools to support measurement and analysis of GPU-accelerated applications. To help developers understand the performance of accelerated applications as a whole, HPCToolkit's measurement and analysis tools attribute metrics to calling contexts that span both CPUs and GPUs. To measure GPU-accelerated applications efficiently, HPCToolkit employs a novel wait-free data structure to coordinate monitoring and attribution of GPU performance. To help developers understand the performance of complex GPU code generated from high-level programming models, HPCToolkit constructs sophisticated approximations of call path profiles for GPU computations. To support fine-grained analysis and tuning, HPCToolkit uses PC sampling and instrumentation to measure and attribute GPU performance metrics to source lines, loops, and inlined code. To supplement fine-grained measurements, HPCToolkit can measure GPU kernel executions using hardware performance counters. To provide a view of how an execution evolves over time, HPCToolkit can collect, analyze, and visualize call path traces within and across nodes. Finally, on NVIDIA GPUs, HPCToolkit can derive and attribute a collection of useful performance metrics based on measurements using GPU PC samples. We illustrate HPCToolkit's new capabilities for analyzing GPU-accelerated applications with several codes developed as part of the Exascale Computing Project.

show abstract

“…Figure 1 presents an overview of INAM. INAM is capabale of profiling and monitoring largescale InfiniBand networks with low overhead [5], profiling GPU and CPU intra-node communication [4], and profiling both MPI and the job scheduler [6]. Therefore, INAM supports profiling HPC/DL applications, MPI communication runtimes, and the job scheduler.…”

Section: Introductionmentioning

confidence: 99%

Cross-layer Visualization and Profiling of Network and I/O Communication for HPC Clusters

Kousha¹,

Anthony²,

Subramoni³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Understanding and visualizing the full-stack performance trade-offs and interplay between HPC applications, MPI libraries, the communication fabric, and the file system is a challenging endeavor. Designing a holistic profiling and visualization method for HPC communication networks is challenging since different levels of communication coexist and interact with each other on the communication fabric. A breakdown of traffic is essential to understand the interplay of different layers along with the application's communication behavior without losing a general view of network traffic.Unfortunately, existing profiling tools are disjoint and either focus on only profiling and visualizing a few levels of the HPC stack, which limits the insights they can provide, or they provide extremely detailed information which necessitates a steep learning curve to understand. We target our profiling tool visualization to provide holistic and real-time insights into HPC communication stacks.In this paper, we propose and implement our visualization methods to enable holistic insight for representing the cross-stack metrics. Moreover, we propose and implement a low-overhead I/O profiling inside the communication library, collect and store the profiling information, and then study the correlation and evaluation of I/O traffic with MPI communication using a crossstack approach by INAM. Through experimental evaluations and use cases, we demonstrate novel benefits of our cross-stack communication analysis in real-time to detect bottlenecks and understand communication performance.

show abstract

Designing a Profiling and Visualization Tool for Scalable and In-depth Analysis of High-Performance GPU Clusters

Cited by 14 publications

References 10 publications

Profiling and Monitoring Deep Learning Training Tasks

Profiling and Monitoring Deep Learning Training Tasks

Measurement and analysis of GPU-accelerated applications with HPCToolkit

Cross-layer Visualization and Profiling of Network and I/O Communication for HPC Clusters

Contact Info

Product

Resources

About