Petascale turbulence simulation using a highly parallel fast multipole method on GPUs

Abstract

This paper reports large-scale direct numerical simulations of homogeneous-isotropic fluid turbulence, achieving sustained performance of 1.08 petaflop/s on gpu hardware using single precision. The simulations use a vortex particle method to solve the Navier-Stokes equations, with a highly parallel fast multipole method (fmm) as numerical engine, and match the current record in mesh size for this application, a cube of 4096 3 computational points solved with a spectral method. The standard numerical approach used in this field is the pseudo-spectral method, relying on the fft algorithm as numerical engine. The particle-based simulations presented in this paper quantitatively match the kinetic energy spectrum obtained with a pseudo-spectral method, using a trusted code. In terms of parallel performance, weak scaling results show the fmm-based vortex method achieving 74% parallel efficiency on 4096 processes (one gpu per mpi process, 3 gpus per node of the tsubame-2.0 system). The fft-based spectral method is able to achieve just 14% parallel efficiency on the same number of mpi processes (using only cpu cores), due to the all-to-all communication pattern of the fft algorithm. The calculation time for one time step was 108 seconds for the vortex method and 154 seconds for the spectral method, under these conditions. Computing with 69 billion particles, this work exceeds by an order of magnitude the largest vortex-method calculations to date.

Acceleration of the Fast Multipole Method on FPGA Devices

“…Output signals have to have operation '+=' that means summation. Vector variables can be described in PG2 as 'xi [3].' Multiplication between vectors like line 7 is converted to an inner product.…”

“…Without a sophisticated algorithm, the computational cost to calculate the gravity between N particles is O(N 2 ), while that of the fast multipole method (FMM) is only O(N) [1], [2]. Recently, the FMM's low communication requirement between computing nodes has become a hot topic since the parallel efficiency of the fast fourier transform often degrades with a large number of processors [3], [4]. Combining an accelerator, such as a graphics processing unit (GPU), with the FMM is one of the promising approaches to speed up the simulation further, as in exa-FMM [5] and gemsFMM [6].…”

A method to compute periodic sums

“…Special purpose hardware such as graphics processors or heterogeneous CPU/GPU architectures also allow the fast computation of finite sums, either via brute force summation [18], or via the mapping of the FMM onto these architectures [19,20,21,22]. Yokota et al [22] favorably compare a large scale FMMbased vortex element computations with a direct numerical simulation via periodic pseudospectral methods.…”

Alya: Multiphysics engineering simulation toward exascale

“…On one hand, current HPC systems lack the computational power, network bandwidth and data storage needed for solving tomorrow's real-world engineering challenges. On the other hand, while emerging peta-scale computing is already a strategic enabler of large-scale simulations in many scientific areas (such as astronomy, biology and chemistry), even the most powerful hardware will fail to deliver on its full potential unless matched with simulation software designed specifically for such environments.Several papers describe the effort of performing large-scale simulations on supercomputers, covering key areas: molecular dynamics [26], mantle convection in solid earth dynamics [3], massive N-body simulations [36], seismic wave propagation [25], weather prediction [1] or fundamentals of turbulence on channels using the vortex method [37]. A similar list can be obtained from the 2014 ACM Gordon Bell Prize in…”

“…Several papers describe the effort of performing large-scale simulations on supercomputers, covering key areas: molecular dynamics [26], mantle convection in solid earth dynamics [3], massive N-body simulations [36], seismic wave propagation [25], weather prediction [1] or fundamentals of turbulence on channels using the vortex method [37]. A similar list can be obtained from the 2014 ACM Gordon Bell Prize in High Performance Computing finalists.…”