INAM: Cross-stack Profiling and Analysis of Communication in MPI-based Applications

Kousha, Pouya; Ram, Kamal Raj Sankarapandian Dayala Ganesh; Kedia, Mansa; Subramoni, Hari; Jain, Arpan; Shafi, Aamir; Panda, Dhabaleswar K.; Dockendorf, Trey; Na, Heechang; Tomko, Karen

doi:10.1145/3437359.3465582

Cited by 9 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…P-strategy Scale (Processes) HPCToolkit [12] S 10k Open|SpeedShop [19] S 256 Score-P [11] I 10k TAU [20] I 16k Timemory [9] I -INAM [21] I 4k Caliper [10] I 256 TABLE I: Summary of a subset of existing popular profiling (P) tools. P-strategy denotes the profiling strategy of tools, including statistical sampling (S) and instrumentation (I).…”

Section: Profilersmentioning

confidence: 99%

“…This can further degrade the performance analysis experience as users must profile once again to improve the data quality. In contrast, instrumentation profilers like Score-P [11], TAU [20], Timemory [9], INAM [21] and Caliper [10] are flexible, customizable, and incur fewer logging overheads. They employ source-code annotation and adopt several compiler mechanisms to record the performance data across several domains (e.g., hardware, application, communication).…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

TinyProf: Towards Continuous Performance Introspection through Scalable Parallel I/O

Fan,

Kesavan,

Petruzza

et al. 2024

ISC High Performance 2024 Research Paper Proceedings (39th International Conference)

View full text Add to dashboard Cite

Performance profiling tools are crucial for HPC specialists to identify performance bottlenecks in parallel codes at various levels of granularity (i.e., across nodes, ranks, and threads). Although numerous sophisticated profiling tools have been developed, achieving scalable performance introspection on large scales remains a challenge. This is particularly evident in efficiently writing profiles to disk during runtime and subsequently reading them with constrained computing resources for posthoc analysis. In this paper, we present TinyProf , a performance introspection framework that tackles I/O-related challenges in profiling performance data at scale. TinyProf 's scalability is attributed to an optimal runtime that consists of three key components: (1) an efficient in-memory data structure that minimizes memory consumption and decreases communication overhead during parallel file I/O; (2) a customizable threephase I/O scheme that generates optimal I/O patterns capable of scaling with high core counts; and (3) a streamlined data format for profiles, which guarantees minimal sizes for profile files. These three techniques instill scalability into the profiler, making it low overhead, even at high process counts (less than 5%). This low overhead makes it possible for the profiler to be run with an application as a default (whenever the application is running)-enabling continuous introspection of performance. We demonstrate the efficiency of our framework using large-scale parallel applications and perform a thorough evaluation against existing systems up to 32k processes.

show abstract