Active Learning-based Automatic Tuning and Prediction of Parallel I/O Performance

Agarwal, Megha; Singhvi, Divyansh; Malakar, Preeti; Byna, Suren

doi:10.1109/pdsw49588.2019.00007

Cited by 16 publications

(7 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, combining optimization in multiple levels and finding the best set of tunable parameters to achieve performance can be cumbersome. That alone is the target of different approaches [2], [3], [7], [8]. The challenge is not only the search space exploration and time required to do it, but also the inter-dependencies between various I/O software layers (i.e., HDF5, MPI-IO, and file systems), and how to detect bottlenecks.…”

Section: Introductionmentioning

confidence: 99%

I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis

Bez

Tang

Xie

et al. 2021

2021 IEEE/ACM Sixth International Parallel Data Systems Workshop (PDSW)

Self Cite

View full text Add to dashboard Cite

Using parallel file systems efficiently is a tricky problem due to inter-dependencies among multiple layers of I/O software, including high-level I/O libraries (HDF5, netCDF, etc.), MPI-IO, POSIX, and file systems (GPFS, Lustre, etc.). Profiling tools such as Darshan collect traces to help understand the I/O performance behavior. However, there are significant gaps in analyzing the collected traces and then applying tuning options offered by various layers of I/O software. Seeking to connect the dots between I/O bottleneck detection and tuning, we propose DXT Explorer, an interactive log analysis tool. In this paper, we present a case study using our interactive log analysis tool to identify and apply various I/O optimizations. We report an evaluation of performance improvement achieved for four I/O kernels extracted from science applications.

show abstract

Section: Introductionmentioning

confidence: 99%

I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis

Bez

Tang

Xie

et al. 2021

2021 IEEE/ACM Sixth International Parallel Data Systems Workshop (PDSW)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The impact of applications' access pattern on performance has been widely studied and reported: the I/O performance observed when accessing a parallel file system depends strongly on the way this access is done [1]- [6]. Many research efforts focus on improving the I/O infrastructure (the parallel file system or an I/O library, for instance) in a way that transparently benefits all applications running on a system [7]- [12]. Still, reaching peak I/O performance often still depends on application tuning [21], which is a more labor-intensive task because of the huge number of different applications that exist.…”

Section: Related Workmentioning

confidence: 99%

“…For those reasons, I/O kernels are sometimes created and made available to the community. Well known examples are HACC-IO 7 and BT-IO 8 . Our MSLIO extends this list by adding a multiscale simulation I/O kernel.…”

Section: A Benchmark Tools and I/o Kernelsmentioning

confidence: 99%

“…A lot of research has focused on improving the I/O stack -file system, I/O forwarding software, I/O libraries, etcto transparently improve performance, i.e. without requiring any modifications in the applications [7]- [12]. Still, it remains important that applications perform I/O in an adequate way in order to achieve the highest performance from the PFS.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

I/O performance of multiscale finite element simulations on HPC environments

Boito

Gomes

Peyrondet

et al. 2022

2022 International Symposium on Computer Architecture and High Performance Computing Workshops (SBAC-PADW)

View full text Add to dashboard Cite

In this paper, we present MSLIO, a code to mimic the I/O behavior of multiscale simulations. Such an I/O kernel is useful for HPC research, as it can be executed more easily and more efficiently than the full simulations when researchers are interested in the I/O load only. We validate MSLIO by comparing it to the I/O performance of an actual simulation, and we then use it to test some possible improvements to the output routine of the MHM (Multiscale Hybrid Mixed) library.

show abstract

“…Application related parameters, storage hardware properties, problem size and concurrency have an important effect on I/O parameters [4]. For example, choosing a larger stripe size and stripe count in the Lustre file system is normally recommended.…”

Section: Introductionmentioning

confidence: 99%

Improving the MPI-IO Performance of Applications with Genetic Algorithm based Auto-tuning

Bagbaba

Wang

2021

2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)

View full text Add to dashboard Cite

Parallel I/O is an essential part of scientific applications running on high performance computing systems. Understanding an application's parallel I/O behaviour and identifying sources of performance bottlenecks require a multi-layer view of the I/O. Typical parallel I/O stack layers offer many tunable parameters that can achieve the best possible I/O performance. However, scientific users do often not have the time nor the experience for investigating the proper combination of these parameters for each application use-case. Auto-tuning can help users by automatically tuning I/O parameters at various layers transparently. In auto-tuning, using naïve strategy, running an application by trying all possible combinations of tunable parameters for all layers of the I/O stack to find the best settings is an exhaustive search through the huge parameter space. This strategy is infeasible because of the long execution times of trial runs. In this paper, we propose a genetic algorithm based parallel I/O auto-tuning approach that can hide the complexity of the I/O stack from users and auto-tune a set of parameter values for an application on a given system to improve the I/O performance. In particular, our approach tests a set of parameters and then, modifies the combination of these parameters for further testing based on the I/O performance. We have validated our model using two I/O benchmarks, namely IOR and MPI-Tile-IO. We achieved an increase in I/O bandwidth of up to 7.74×over the default parameters for IOR and 5.59×over the default parameters for MPI-Tile-IO.

show abstract

Active Learning-based Automatic Tuning and Prediction of Parallel I/O Performance

Cited by 16 publications

References 20 publications

I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis

I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis

I/O performance of multiscale finite element simulations on HPC environments

Improving the MPI-IO Performance of Applications with Genetic Algorithm based Auto-tuning

Contact Info

Product

Resources

About