Modern gyrokinetic particle-in-cell simulation of fusion plasmas on top supercomputers

Abstract

The Gyrokinetic Toroidal Code at Princeton (GTC-P) is a highly scalable and portable particle-incell (PIC) code. It solves the 5D Vlasov-Poisson equation featuring efficient utilization of modern parallel computer architectures at the petascale and beyond. Motivated by the goal of developing a modern code capable of dealing with the physics challenge of increasing problem size with sufficient resolution, new thread-level optimizations have been introduced as well as a key additional domain decomposition. GTC-P's multiple levels of parallelism, including inter-node 2D domain decomposition and particle decomposition, as well as intra-node shared memory partition and vectorization have enabled pushing the scalability of the PIC method to extreme computational scales. In this paper, we describe the methods developed to build a highly parallelized PIC code across a broad range of supercomputer designs. This particularly includes implementations on heterogeneous systems using NVIDIA GPU accelerators and Intel Xeon Phi (MIC) co-processors and performance comparisons with state-of-the-art homogeneous HPC systems such as Blue Gene/Q. New discovery science capabilities in the magnetic fusion energy application domain are enabled, including investigations of Ion-Temperature-Gradient (ITG) driven turbulence simulations with unprecedented spatial resolution and long temporal duration. Performance studies with realistic fusion experimental parameters are carried out on multiple supercomputing systems spanning a wide range of cache capacities, cache-sharing configurations, memory bandwidth, interconnects and network topologies. These performance comparisons using a realistic discovery-sciencecapable domain application code provide valuable insights on optimization techniques across one of the broadest sets of current high-end computing platforms worldwide. arXiv:1510.05546v1 [cs.DC] 19 Oct 2015 sustainable supply of energy on Earth. Magnetic confinement fusion devices in which very high temperature plasma is confined in a tokamak, a donut-shaped vacuum vessel, is one of the most promising approaches to fusion reactors. In such a system, the interplay between the complex trajectories of individual charged particles and the collective effects arising from the long range nature of electromagnetic forces lead to a wide range of waves, oscillations, and instabilities. These include larger-scale (macro) instabilities that produce rapid topological changes in the device resulting in a catastrophic loss of fusion power, as well as smaller-scale (micro) instabilities that gradually leak energy and thus affect performance and economic viability. Turbulent fluctuations ("microturbulence") are instabilities caused by the unavoidable spatial variations (gradients) in a plasma system. They can significantly increase the transport rate of heat, particles, and momentum across the confining magnetic field and severely limit the energy confinement time for a given machine size. High-fidelity predictive numerical experiments based on 5D gyrokinet...

Sparse grid techniques for particle-in-cell schemes

“…Its numerous desirable features explain in large part its success -it is conceptually simple, readily parallelizable, relatively robust, and can incorporate a wide variety of physical phenomena. Even so, PIC simulations of the complex, three-dimensional systems that arise in modern plasma physics applications still require many hours on a massively parallel machine [15,16,17,18,41].…”

Jefipic: A 3-D Particle-in-Cell Method Based on Jefimenko's Equations on Gpu

“…Further, Marder [4] and Villasenor [5] amended the error of the electric field divergence, and Birdsall [6] and Vahedi [7] introduced the Monte Carlo collision into PIC, which made PIC step into the practical application. Nowadays, PIC is often used to simulate the controlled/laser thermonuclear fusion [8], [9], nuclear explosion effects [10], [11], [12], space physics [13], or design the vacuum electronic devices [14], [15], [16].…”

“…(1) ~ (2) as well as the current continuity equation Eq. (8). The rest two divergence equations Eqs.…”

Machine learning accelerated particle-in-cell plasma simulations

“…Simulations of macroscopic systems which employ kinetic formulations of the physics model quickly become prohibitively expensive, even for leadership class, pre-exascale compute facilities. Researchers are therefore constantly implementing advanced parallelization and optimization methods, often tailored to specific target hardware architectures, in order to speed up such codes [5,6,7,8,9]. In this contribution we are reporting on the development of a machine learning aided amortized solver that accelerates PIC simulations.…”