Performance characterization of irregular I/O at the extreme scale

Abstract

This paper reports our experience with irregular I/O and describes lessons learned when running applications with such I/O on supercomputers at the extreme scale. Specifically, we study how irregularities in I/O patterns (i.e., irregular amount of data written per process at each I/O step) in scientific simulations can cause increasing I/O times and substantial loss in scalability. To this end, we quantify the impact of irregular I/O patterns on the I/O performance of scientific applications at the extreme scale by statistically modeling the irregular I/O behavior of two scientific applications: the Monte Carlo application QMCPack and the adaptive mesh refinement application ENZO. For our testing, we feed our model into I/O kernels of two well-known I/O data models (i.e., ADIOS and HDF) to measure the performance of the two applications' I/O under different I/O settings. Empirically, we show how the growing data sizes and the irregular I/O patterns in these applications are both relevant factors impacting performance.

Proposal of an open-source computational toolbox for solving PDEs in the context of chemical reaction engineering using FEniCS and complementary components

“…Next, the simulation is run, and the results computed by the FEniCS may be stored for visualization or reused later in Python-FEniCS computations. The results are stored in the HDF5 format, which is designed to support I/O parallel operations and is useful for high-performance computing (HPC) ( Herbein et al., 2016 ; Petter Langtangen and Logg, 2017 ; Soumagne and Biddiscombe, 2011 ).…”