I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis

Abstract

Using parallel file systems efficiently is a tricky problem due to inter-dependencies among multiple layers of I/O software, including high-level I/O libraries (HDF5, netCDF, etc.), MPI-IO, POSIX, and file systems (GPFS, Lustre, etc.). Profiling tools such as Darshan collect traces to help understand the I/O performance behavior. However, there are significant gaps in analyzing the collected traces and then applying tuning options offered by various layers of I/O software. Seeking to connect the dots between I/O bottleneck detection and tuning, we propose DXT Explorer, an interactive log analysis tool. In this paper, we present a case study using our interactive log analysis tool to identify and apply various I/O optimizations. We report an evaluation of performance improvement achieved for four I/O kernels extracted from science applications.

Drishti: Guiding End-Users in the I/O Optimization Journey

“…2 depicts the Drishti report for this execution, and Fig. 3 confirms that the applied optimizations (refer to [5]) changed the application I/O behavior, avoiding common pitfalls. 2) Domain Decomposition Use Case: In previous work [5], we investigated the end-to-end (E2E) [44] domain decomposition I/O kernel.…”

“…A plethora of related work [9]- [14] has exposed some of the most common root causes of I/O performance bottlenecks originating from the application down to the file system. Though some issues are intrinsic to how the application was modeled and coded, others reflect the complexity of correctly tuning the exposed parameters or the interplay between factors in the I/O software and hardware stack [5], [15]- [18].…”

“…Efforts such as IOMiner [21], UMAMI [12], and TOKIO [22] provide a framework to analyze combined metrics seeking to detect root causes of I/O problems. DXT Explorer [5] approaches the problem visually and interactively but does not provide guidance on how to fix the issues. On the other hand, auto-tuning approaches [15], [17], [18], [23] attempt relieving the burden, on users, of finding and fixing performance issues due to mis-configuration, but they are application-specific and time-consuming.…”

“…Its library [42] provides a reference implementation of the standard using file formats such as HDF5 [1] and ADIOS [43]. In previous work [5], where we used openPMD-api 0.14.1 with HDF5 backend we noticed performance issues caused by a now-fixed bug in HDF5, which made collective metadata operations become individual (thus, issuing a lot of small requests), mis-aligned requests, and small data writes and reads. Fig.…”

“…The interdependencies among layers of I/O software, including highlevel parallel I/O libraries (e.g., HDF5 [1], PnetCDF [2]) interfaces (e.g., MPI-IO, POSIX, STDIO), and file systems (e.g., Lustre [3], GPFS [4]) makes this a non-trivial task that often requires an I/O expert in the loop. Despite numerous efforts from the community to collect I/O profiles or traces and propose new optimization and tuning solutions to improve performance, there is still a gap between profiling and tuning [5]. The main challenge in transforming I/O profiles into meaningful information is to detect root causes of I/O bottlenecks and mapping them into actionable items so an end-user can receive guidance on tuning I/O performance.…”