Scalable I/O tracing and analysis

Abstract

As supercomputer performance approached and then surpassed the petaflop level, I/O performance has become a major performance bottleneck for many scientific applications. Several tools exist to collect I/O traces to assist in the analysis of I/O performance problems. However, these tools either produce extremely large trace files that complicate performance analysis, or sacrifice accuracy to collect high-level statistical information. We propose a multi-level trace generator tool, ScalaIOTrace, that collects traces at several levels in the HPC I/O stack. ScalaIOTrace features aggressive trace compression that generates trace files of near constant size for regular I/O patterns and orders of magnitudes smaller for less regular ones. This enables the collection of I/O and communication traces of applications running on thousands of processors.Our contributions also include automated trace analysis to collect selected statistical information of I/O calls by parsing the compressed trace on-the-fly and time-accurate replay of communication events with MPI-IO calls. We evaluated our approach with

Boosting Application-Specific Parallel I/O Optimization Using IOSIG

“…There are a few tools focusing on tracing data I/O events, like HPCT-IO [5], LANL-Trace [8], IOT [7], and ScalaIOTrace [9], but these tools do not provide enough comprehensive trace analysis or data access representation.…”

“…However, there is a serious lack of tools for analyzing parallel I/O performance in a comprehensive manner and for converting the analyzed data into information that optimization techniques can use. The existing I/O analysis tools [5] [6] [7] [8] [9] have limited scope of I/O characterization. Few of these tools [5] [8] collect a lot of trace information about I/O calls and leave it for programmers to understand.…”

“…Without the analysis step, although some I/O traces are available, they just sit idle in some server and are not useful for improving the efficiency. A few other tools [6] [7] [9] provide partial understanding of I/O behavior but also require programmer involvement in performing optimizations. The latter category of tools aims towards reducing overhead and resource requirement in collecting information about I/O calls by retrieving few details and infrequently.…”

ScalaTrace: Tracing, Analysis and Modeling of HPC Codes at Scale

“…Traces can further be extrapolated in the dimension of number of tasks (nodes) to assess communication scalability and assist procurement decisions for future HPC installations. Further information about ScalaTrace can be found elsewhere [7,9,8,11].…”

Automated Tracing of I/O Stack

“…INS-OP [15] is also a dynamic instrumentation tool that applies transformations to reduce the overheads in the instrumentation code. In [27], Vijayakumar et al propose an I/O tracing approach that combines aggressive trace compression. However, their strategy does not provide flexibility in terms of target metric specification.…”