PIONEER: A Solution to Parallel I/O Workload Characterization and Generation

He, Weiping; Du, David Hung-Chang; Narasimhamurthy, Sai

doi:10.1109/ccgrid.2015.32

Cited by 5 publications

(2 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In general, synthesizing representative I/O workloads requires some sort of model or characterization of the target application's I/O behavior. In most existing research, HPC application I/O characterizations are generated by first tracing the I/O operations of a target application and performing an in-depth analysis of this trace offline [5,14,18,30,37]. In contrast, Darshan I/O characterizations are automatically generated at runtime.…”

Section: Synthesizing Hpc I/o Workloadsmentioning

confidence: 99%

“…This approach is interesting because it can yield representative I/O workloads for a given application with no human intervention. He et al present PIONEER [18], a parallel I/O workload characterization and generation framework, which attempts to address open problems in parallel I/O workload modeling, such as interprocess correlations and I/O library request dependencies. The PIONEER approach is to analyze a workload's trace files offline in order to determine interprocess correlations and to create a generic workload presentation that is used by all workload processes to regenerate the workload.…”

Section: Synthesizing Hpc I/o Workloadsmentioning

confidence: 99%

See 1 more Smart Citation

Techniques for modeling large-scale HPC I/O workloads

Snyder

Carns

Latham

et al. 2015

Proceedings of the 6th International Workshop on Performance Modeling, Benchmarking, and Simulation of High Performance Computi

View full text Add to dashboard Cite

Accurate analysis of HPC storage system designs is contingent on the use of I/O workloads that are truly representative of expected use. However, I/O analyses are generally bound to specific workload modeling techniques such as synthetic benchmarks or trace replay mechanisms, despite the fact that no single workload modeling technique is appropriate for all use cases. In this work, we present the design of IOWA, a novel I/O workload abstraction that allows arbitrary workload consumer components to obtain I/O workloads from a range of diverse input sources. Thus, researchers can choose specific I/O workload generators based on the resources they have available and the type of evaluation they wish to perform. As part of this research, we also outline the design of three distinct workload generation methods, based on I/O traces, synthetic I/O kernels, and I/O characterizations. We analyze and contrast each of these workload generation techniques in the context of storage system simulation models as well as production storage system measurements. We found that each generator mechanism offers varying levels of accuracy, flexibility, and breadth of use that should be considered before performing I/O analyses. We also recommend a set of best practices for HPC I/O workload modeling based on challenges that we encountered while performing our evaluation.

show abstract