Communication-overlap techniques for improved strong scaling of gyrokinetic Eulerian code beyond 100k cores on the K-computer

Abstract

Plasma turbulence research based on five-dimensional (5D) gyrokinetic simulations is one of the most critical and demanding issues in fusion science. To pioneer new physics regimes both in problem sizes and in timescales, an improvement of strong scaling is essential. Overlap of computations and communications using non-blocking MPI communication schemes is a promising approach to improving strong scaling, but it often fails on practical applications with conventional MPI libraries. In this work, this classical issue is resolved by developing communication-overlap techniques with additional MPI support for non-blocking communication routines and with heterogeneous OpenMP threads, which work even on conventional MPI libraries and network hardware. These techniques dramatically improved the parallel efficiency of a gyrokinetic toroidal 5D Eulerian code GT5D on the K-computer, which has a dedicated network, and on the Helios system which has a commodity network. On the K-computer, excellent strong scaling was achieved beyond 100k cores whilst keeping a sustained performance of false~ 10% ( false~ 307 TFlops using 196,608 cores), and simulations for next-generation large-scale fusion experiments are significantly accelerated. This performance is 16 false× sped up compared with the maximum performance reported at the 2011 International Conference for High Performance Computing, Networking, Storage and Analysis ( false~ 19 TFlops using 16,384 cores of the BX900 cluster) (Idomura, 2011).

Computation-Communication Overlap Techniques for Parallel Spectral Calculations in Gyrokinetic Vlasov Simulations

“…The transpose communications required for parallel 2D FFTs are implemented by using a simple blocking collective communi-cation, which makes implementation easy and the use of specific algorithms possible, like collective communications optimized for the K computer [11]. Second, transpose communications and FFT computations are overlapped by employing a communication thread in a hybrid parallel model of Message Passing Interface (MPI) and Open Multi-Processing (OpenMP), which enables overlaps of computations and blocking communications as well as non-blocking communications [12]. Masking of communication costs significantly improves strong scaling of the perpendicular-space parallelization.…”

Improved strong scaling of a spectral/finite difference gyrokinetic code for multi-scale plasma turbulence

“…Since operations of 1D FFT are parallelized by OpenMP threads, we employ the thread-safe 1D FFTs by means of FFTW [14]. The idea of computation-communication overlaps with a communication thread has already been tested before [15][16][17]. However, for efficiently masking the communication cost by applying the pipelined overlaps, careful implementations are required: rearrangements of multiple computation kernels to keep computations enough to mask communications, a proper choice of the pipeline length (finer pipelining reduces unoverlapped parts at the beginning and the end of pipelining but increases latencies of MPI), and a regulation of the task granularity for thread parallelization (finer granularity reduces load imbalance on OpenMP threads but increases scheduling overheads).…”

Progress of Full-<i>f</i> Gyrokinetic Simulation Toward Reactor Relevant Numerical Experiments

“…The domain decomposition model is implemented using a hybrid parallelization model consisting of multi-layer MPI communicators and multi-core OpenMP parallelization. In addition, a novel computation and communication overlap technique [20] is developed using communication threads, which are implemented with a heterogeneous OpenMP programing model. The strong scaling of GT5D is dramatically improved by this latency hiding technique, and on the K-computer, an excellent strong scaling is achieved up to ∼ 0.6 million cores (see Fig.…”