Drishti: Guiding End-Users in the I/O Optimization Journey

Abstract

The complex software and hardware I/O stack of HPC platforms makes it challenging for end-users to obtain superior I/O performance and to understand the root causes of I/O bottlenecks they encounter. Despite the continuous efforts from the community to profile I/O performance and propose new optimization techniques and tuning options for improving the performance, there is still a translation gap between profiling and tuning. In this paper, we propose Drishti, a solution to guide scientists in optimizing I/O in their applications by detecting typical I/O performance pitfalls and providing recommendations. We illustrate its applicability in two case studies and evaluate its robustness and performance by summarizing the issues detected in over a hundred thousand Darshan logs collected on the Cori supercomputer at the National Energy Research Scientific Computing Center (NERSC). Drishti can empower end-users and guide them in the I/O optimization journey by shedding some light on everyday I/O performance pitfalls and how to fix them.

Automated Network Performance Characterization for HPC Systems

“…This will provide a comprehensive and interactive analysis for various use cases [6,28]. For example, user hints such as Drishti [5] as well as bottleneck detection, pattern analysis, and performance prediction can be provided, leading to valuable insights for understanding the performance of emerging workloads.…”

Enabling Agile Analysis of I/O Performance Data with PyDarshan

“…• Use Case 1: Enhancing single job summaries with HTML reports and modular templates for more interactivity using a large-scale run of the E3SM [20] climate code in Section 3.1 • Use Case 2: Enabling custom analysis tools building on top of Darshan using the examples of DXT Explorer [11] and Drishti [10] • Use Case 3: Customizing I/O analysis of workflows using the example of ATLAS AthenaMP, a high-energy physics simulation in Section 3.3 • Use Case 4: Enabling the analysis of large bodies of Darshan logs with hundreds of thousands of jobs on the Cori and Theta supercomputers in Section 3.4…”