Improved strong scaling of a spectral/finite difference gyrokinetic code for multi-scale plasma turbulence

Maeyama, Shinya; Watanabe, T.‐H.; Idomura, Yasuhiro; Nakata, M.; Nunami, M.; Ishizawa, A.

doi:10.1016/j.parco.2015.06.001

Cited by 13 publications

(13 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To reveal the physical process of the cross-scale interactions between electron and ion scale turbulence, we have carried out electromagnetic gyrokinetic simulations of multi-scale ETG/ITG turbulence with realistic hydrogen-to-electron mass ratio and β value [4]. The extensive simulations were realized by developing a massively parallel algorithm of the gyrokinetic simulation code GKV [5]. The multi-scale simulations demonstrated that strong suppression of electron-scale transport by ion-scale turbulence as reported previously [1,2].…”

mentioning

confidence: 99%

Cross-Scale Interactions between Electron and Ion Scale Turbulence in a Tokamak Plasma

et al. 2015

View full text Add to dashboard Cite

Multiscale gyrokinetic turbulence simulations with the real ion-to-electron mass ratio and β value are realized for the first time, where the β value is given by the ratio of plasma pressure to magnetic pressure and characterizes electromagnetic effects on microinstabilities. Numerical analysis at both the electron scale and the ion scale is used to reveal the mechanism of their cross-scale interactions. Even with the real-mass scale separation, ion-scale turbulence eliminates electron-scale streamers and dominates heat transport, not only of ions but also of electrons. Suppression of electron-scale turbulence by ion-scale eddies, rather than by long-wavelength zonal flows, is also demonstrated by means of direct measurement of nonlinear mode-to-mode coupling. When the ion-scale modes are stabilized by finite-β effects, the contribution of the electron-scale dynamics to the turbulent transport becomes non-negligible and turns out to enhance ion-scale turbulent transport. Damping of the ion-scale zonal flows by electron-scale turbulence is responsible for the enhancement of ion-scale transport.

show abstract

mentioning

confidence: 99%

Cross-Scale Interactions between Electron and Ion Scale Turbulence in a Tokamak Plasma

et al. 2015

View full text Add to dashboard Cite

show abstract

“…In other words, the domain decomposition in k x direction results in extra transpose communications for boundary computations. The details about other communication schemes can be found in the work of Maeyama et al…”

Section: Code Descriptionmentioning

confidence: 99%

“…As the first step of porting GKV code, we revisit the communication pattern in convolution operation. In the original version, the parallel FFT is carried out in the way shown in Algorithm 1 (duplicated from Table 1, left column, in the work of Idomura et al with minor modifications in notations). Here, the subscripts denote that all the data points in those directions are held by each MPI process.…”

Section: Code Descriptionmentioning

confidence: 99%

“…The increased cost of finite difference kernel is induced by the increased halo communication costs. To mitigate the communication costs, we apply the same technique as in the work of Maeyama et al, namely pipelining the kernels (see Section 5.1.2).…”

Section: Performance Measurements and Hotspotsmentioning

confidence: 99%

“…So as to mask the communication cost, the pipeline‐based computation and communication overlapping method was proposed for 5D gyrokinetic simulation codes based on the finite difference method and the spectral method . An improved strong scaling up to 600 k cores was demonstrated on a conventional CPU‐based supercomputer . It is, however, still questionable to have a similar scalability on state‐of‐the art supercomputing systems employing accelerators, where the computational power is enormously increased while the improvement of the communication speed is limited.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Overlapping communications in gyrokinetic codes on accelerator‐based platforms

Asahi

Latu

Bigot

et al. 2019

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Communication and computation overlapping techniques have been introduced in the five-dimensional gyrokinetic codes GYSELA and GKV. In order to anticipate some of the exa-scale requirements, these codes were ported to the modern accelerators, Xeon Phi KNL and Tesla P 100 GPU. On accelerators, a serial version of GYSELA on KNL and GKV on GPU are respectively 1.3× and 7.4× faster than those on a single Skylake processor (a single socket). For the scalability, we have measured GYSELA performance on Xeon Phi KNL from 16 to 512 KNLs (1024 to 32k cores) and GKV performance on Tesla P 100 GPU from 32 to 256 GPUs. In their parallel versions, transpose communication in semi-Lagrangian solver in GYSELA or Convolution kernel in GKV turned out to be a main bottleneck. This indicates that in the exa-scale, the network constraints would be critical. In order to mitigate the communication costs, the pipeline and task-based overlapping techniques have been implemented in these codes. The GYSELA 2D advection solver has achieved a 33% to 92% speed up, and the GKV 2D convolution kernel has achieved a factor of 2 speed up with pipelining. The task-based approach gives 11% to 82% performance gain in the derivative computation of the electrostatic potential in GYSELA.We have shown that the pipeline-based approach is applicable with the presence of symmetry, while the task-based approach can be applicable to more general situations. KEYWORDSoverlap, semi-Lagrangian, spectral, Tesla P100 GPU, transpose communication, Xeon Phi KNL INTRODUCTIONIt is known that turbulence in a magnetic confined fusion plasma exhibits strong anisotropies in parallel and perpendicular directions to magnetic fields. In the parallel direction along the magnetic field line, the characteristic scale is the machine size (ie, the order of meter), while in the perpendicular direction, the characteristic size is down to a tiny Larmor radius scale (ie, the order of centimeter). In numerical simulations, higher resolution is needed in the perpendicular direction than in the parallel direction. It is therefore reasonable to parallelize such a simulation in the perpendicular directions since they require a large numbers of grid points. There are several algorithms used in this field, which need a global data structure in the perpendicular directions, like spectral or semi-Lagrangian methods. In order to gather distributed data, we need the so-called transpose communication. However, this type of communication is relatively demanding.So as to mask the communication cost, the pipeline-based computation and communication overlapping method was proposed for 5Dgyrokinetic simulation codes based on the finite difference method 1 and the spectral method. 2,3 An improved strong scaling up to 600 k cores was demonstrated on a conventional CPU-based supercomputer. 1,3 It is, however, still questionable to have a similar scalability on state-of-the art supercomputing systems employing accelerators, where the computational power is enormously increased while the improvement of the ...

show abstract