Multi‐GPU performance optimization of a computational fluid dynamics code using OpenACC

Abstract

Summary This article investigates the multi‐GPU performance of a 3D buoyancy driven cavity solver using MPI and OpenACC directives on multiple platforms. The article shows that decomposing the total problem in different dimensions affects the strong scaling performance significantly for the GPU. Without proper performance optimizations, it is shown that 1D domain decomposition scales poorly on multiple GPUs due to the noncontiguous memory access. The performance using whatever decompositions can be benefited from a series of performance optimizations in the article. Since the buoyancy driven cavity code is communication‐bounded on the clusters examined, a series of optimizations both agnostic and tailored to the platforms are designed to reduce the communication cost and improve memory throughput between hosts and devices efficiently. First, the parallel message packing/unpacking strategy developed for noncontiguous data movement between hosts and devices improves the overall performance by about a factor of 2. Second, transferring different data based on the stencil sizes for different variables further reduces the communication overhead. These two optimizations are general enough to be beneficial to stencil computations having ghost exchanges. Third, GPUDirect is used to improve the communication on clusters which have the hardware and software support for direct communication between GPUs without staging the memory of CPU. Finally, overlapping the communication and computations is shown to be not efficient on multi‐GPUs if only using MPI or MPI+OpenACC. Although we believe our implementation has revealed enough communication and computation overlap, the actual running does not utilize the overlap well due to a lack of enough asynchronous progression.

A novel acceleration method for molecular dynamics of crystal silicon on GPUs using OpenACC

“…At present, there are researches on using OpenACC for application optimization on GPUs. [24][25][26] To verify the performance, portability and productivity of the proposed scheme on SW26010 heterogeneous many-core processor on GPU, we focus on the OpenACC acceleration of MD simulation of silicon crystal on GPUs. The main work of this article includes:…”

“…OpenACC 23 is a directive‐based programming model for acceleration devices, which has been widely supported by the industry. At present, there are researches on using OpenACC for application optimization on GPUs 24‐26 . To verify the performance, portability and productivity of the proposed scheme on SW26010 heterogeneous many‐core processor on GPU, we focus on the OpenACC acceleration of MD simulation of silicon crystal on GPUs.…”

An Improved Framework of GPU Computing for CFD Applications on Structured Grids using OpenACC

“…V1: Pack/Unpack. The goal of this optimization is to improve the memory throughput and reduce the communication cost if the required data are not located sequentially in memory [23]. As Fortran is a column-majored language, the first index i of a matrix A(i, j, k) denotes the fastest change.…”

“…Using ssspnt, different problems, platforms and different scalings can be compared more easily. Similar to Ref [23], every time in this paper is measured consecutively for at least three instances. The difference for each time point is smaller than 1% (usually less than 1 s out of more than 120 s).…”

“…Xue et al [22] applied multiple GPUs for a complicated CFD code on two different platforms but the speedup was not satisfactory (only up to 4× on a NVIDIA P100 GPU), even with some performance optimizations. Also, Xue et al [23] investigated the multi-GPU performance and its performance optimization of a 3D buoyancy-driven cavity solver using MPI and OpenACC directives. They showed that decomposing the total problem in dif-ferent dimensions affects the strong scaling performance significantly when using multiple GPUs.…”