PittPack: An open-source Poisson’s equation solver for extreme-scale computing with accelerators

Abstract

We present a parallel implementation of a direct solver for the Poisson's equation on extreme-scale supercomputers with accelerators. We introduce a chunkedpencil decomposition as the domain-decomposition strategy to distribute work among processing elements to achieve superior scalability at large number of accelerators. Chunked-pencil decomposition enables overlapping nodal communication and data transfer between the central processing units (CPUs) and the graphics processing units (GPUs). Second, it improves data locality by keeping neighboring elements in adjacent memory locations. Third, it allows usage of shared-memory for certain segments of the algorithm when possible, and last but not least, it enables contiguous message transfer among the nodes. Two different communication patterns are designed. The fist pattern aims to fully overlap the communication with data transfer and designed for speedup of overall turnaround time, whereas the second method concentrates on low memory usage and is more network friendly for computations at extreme scale. To ensure software portability, we interleave OpenACC with MPI in the software. The numerical solution and its formal second order of accuracy is verified using method of manufactured solutions for various combinations of boundary conditions. Weak scaling analysis is performed using up to 1.1 trillion Cartesian mesh points using 16384 GPUs on a petascale leadership class supercomputer.

GPU acceleration of CaNS for massively-parallel direct numerical simulations of canonical fluid flows

“…Despite having a similar interface to the FFTW library used in the CPU version, cuFFT does not support the family of real-to-real transforms implemented in FFTW. Therefore, the different real-to-real transforms must be implemented by pre-and postprocessing FFTs [21,17,22]. Some of these transforms have been implemented in the GPU version, namely the standard fast discrete sine and cosine transforms DCT-II and DST-II (and the corresponding inverse transforms, DCT-III and DST-III), following the low-storage approach of Makhoul [21].…”