Computation-Communication Overlap Techniques for Parallel Spectral Calculations in Gyrokinetic Vlasov Simulations

Abstract

One of the important phenomena in magnetically-confined fusion plasma is plasma turbulence, which causes particle and heat transport and degrades plasma confinement. To address multi-scale turbulence including temporal and spatial scales of electrons and ions, we extend our gyrokinetic Vlasov simulation code GKV to run efficiently on peta-scale supercomputers. A key numerical technique is the parallel Fast Fourier Transform (FFT) required for parallel spectral calculations, where masking of the cost of inter-node transpose communications is essential to improve strong scaling. To mask communication costs, computation-communication overlap techniques are applied for FFTs and transpose with the help of the hybrid parallelization of message passing interface and open multi-processing. Integrated overlaps including whole spectral calculation procedures show better scaling than simple overlaps of FFTs and transpose. The masking of communication costs significantly improves strong scaling of the GKV code, and makes substantial speed-up toward multi-scale turbulence simulations.

Improved strong scaling of a spectral/finite difference gyrokinetic code for multi-scale plasma turbulence

“…achieves α ∼99.99994%, which is improved by an order of magnitude compared with the previous results (∼99.9998% in Ref. [18]). This is because the implementation of the species parallelization increases the number of available cores, and the present optimizations improve scalability by reducing and masking the communication cost.…”

“…To improve the strong scaling, it is important that all computations of the spectral calculations (not only forward and backward FFTs but also real space calculations, buffer copies, and data expansion/truncation for the de-aliasing) are overlapped. Therefore, we implemented an integrated overlaps where whole spectral calculation procedures are parallelized by OpenMP and data transpose is overlapped with them [18]. At the same time, we use dynamic scheduling and asynchronous parallelization of OpenMP threads so that the master thread also join the computations after the end of communications.…”

“…Since computations and communications are overlapped, total elapsed time for spectral calculations and for finite difference calculations are plotted. The results are compared with the ideally-overlapped cost estimation given by Maeyama et al [18] C estimation overlapped = C + T N threads ,…”

Validation metrics for turbulent plasma transport

“…The most accurate solutions are obtained through nonlinear, initial value simulations of either the gyrokineticMaxwell equations or their fluid variants, analogous to direct numerical simulation (DNS) of neutral fluid turbulence. A variety of such codes have been developed, which can generally be divided between those taking a continuum approach [52][53][54][55][56] and those that use particle-in-cell methods. [57][58][59][60][61][62] As in neutral fluid DNS, both approaches initialize the simulation with some very small amplitude fluctuations, which first grow exponentially at the linear growth rate(s) of the instabilities being considered (the "linear" phase), and then saturate at a finite amplitude set by the balance of these linear drives and nonlinear couplings between different wavenumbers (the "saturated" phase).…”

Overlapping communications in gyrokinetic codes on accelerator‐based platforms

“…So as to mask the communication cost, the pipeline‐based computation and communication overlapping method was proposed for 5D gyrokinetic simulation codes based on the finite difference method and the spectral method . An improved strong scaling up to 600 k cores was demonstrated on a conventional CPU‐based supercomputer .…”