Accelerating Multigrid-based Hierarchical Scientific Data Refactoring on GPUs

Abstract

Rapid growth in scientific data and a widening gap between computational speed and I/O bandwidth makes it increasingly infeasible to store and share all data produced by scientific simulations. Multigrid-based hierarchical data refactoring is a class of promising approaches to this problem. These approaches decompose data hierarchically; the decomposed components can then be selectively and intelligently stored or shared, based on their relative importance in the original data. Efficient data refactoring design is one key to making these methods truly useful. In this paper, we describe highly optimized data refactoring kernels on GPU accelerators that are specialized for refactoring scientific data. We demonstrate that our optimized design can achieve 45.42 TB/s aggregated data refactoring throughput when using 4,096 GPUs of the Summit supercomputer. Finally, we showcase our optimized design by applying it to a large-scale scientific visualization workflow and the MGARD lossy compression software.

Accelerating Multigrid-based Hierarchical Scientific Data Refactoring on GPUs

“…• OPT: Our GPU data refactoring, which uses our novel processing kernels (i.e., GPK, LPK, and IPK) proposed in our previous work [22].…”

“…In the state-of-the-art design [15], the computed coefficients are first copied to the workspace before they are used for computing corrections, which prohibits out-of-place computing unless additional memory space is used. Our previous work leverages kernel fusion [22] to merges the copy of the coefficients with the first mass-trans matrix multiplication, so that it enables us to use LPK out-of-place compute without a significant increase in memory footprint. However, one drawback with our previous design is the degraded memory access efficiency for GPK since accessing nodes in coarser grids leads to larger strided memory accesses.…”

“…Namely, our hierarchical batch optimization involves two levels of batch optimization: 1) locality batch optimization; and 2) dimensional batch optimization. Batch optimization exploiting data locality in data refactoring on GPUs was proposed in our previous work [22], which is well adopted in GPK, LPK, and IPK. However, it is challenging to use locality batch optimization beyond three-dimensional data refactoring due to limited share memory space on .…”

“…As dense multi-GPU systems are commonly used in modern HPC systems, we extend the GPU data refactoring [22] to take advantage of multi-GPU systems. Dense multi-GPU systems refers to systems where multiple GPUs are equipped in each single computing node of supercomputers with fast interconnect such as NVLiks or PCIe.…”