Parthenon—a performance portable block-structured adaptive mesh refinement framework

Abstract

On the path to exascale the landscape of computer device architectures and corresponding programming models has become much more diverse. While various low-level performance portable programming models are available, support at the application level lacks behind. To address this issue, we present the performance portable block-structured adaptive mesh refinement (AMR) framework Parthenon, derived from the well-tested and widely used Athena++ astrophysical magnetohydrodynamics code, but generalized to serve as the foundation for a variety of downstream multi-physics codes. Parthenon adopts the Kokkos programming model, and provides various levels of abstractions from multidimensional variables, to packages defining and separating components, to launching of parallel compute kernels. Parthenon allocates all data in device memory to reduce data movement, supports the logical packing of variables and mesh blocks to reduce kernel launch overhead, and employs one-sided, asynchronous MPI calls to reduce communication overhead in multi-node simulations. Using a hydrodynamics miniapp, we demonstrate weak and strong scaling on various architectures including AMD and NVIDIA GPUs, Intel and AMD x86 CPUs, IBM Power9 CPUs, as well as Fujitsu A64FX CPUs. At the largest scale on Frontier (the first TOP500 exascale machine), the miniapp reaches a total of 1.7 × 1013 zone-cycles/s on 9216 nodes (73,728 logical GPUs) at [Formula: see text] weak scaling parallel efficiency (starting from a single node). In combination with being an open, collaborative project, this makes Parthenon an ideal framework to target exascale simulations in which the downstream developers can focus on their specific application rather than on the complexity of handling massively-parallel, device-accelerated AMR.

Bridging Scales in Black Hole Accretion and Feedback: Magnetized Bondi Accretion in 3D GRMHD

“…Our numerical scheme utilizes the GRMHD code KHARMA, 6 a performance-portable C++ implementation based on iharm3D (Prather et al 2021); iharm3D is itself an extension of HARM, an efficient second-order conservative finite-volume scheme for solving magnetohydrodynamic (MHD) equations on Eulerian meshes in stationary curved spacetimes (Gammie et al 2003). KHARMA offers a more flexible, portable, and scalable implementation suitable for multiple uses, by leveraging the Parthenon Adaptive Mesh Refinement Framework and the Kokkos programming model (Trott et al 2022;Grete et al 2023).…”

The Launching of Cold Clouds by Galaxy Outflows. V. The Role of Anisotropic Thermal Conduction

“…Unlike in our previous papers, the simulations here were conducted using the performance-portable, open-source ATHE-NAPK code. 3 ATHENAPK implements various finite volume hydro-and magnetohydrodynamics algorithms in addition to PARTHENON (Grete et al 2022), which itself is a performanceportable AMR framework based on ATHENA++ (Stone et al 2020), K-ATHENA (Grete et al 2021), and KOKKOS (Edwards et al 2014;Trott et al 2021). ATHENAPK allows for efficient simulations with AMR on various devices including GPUs from different vendors and has demonstrated scalability with >90% parallel efficiency on up to 73,728 GPUs.…”

Early experiences on the OLCF Frontier system with AthenaPK and Parthenon‐Hydro

“…Overall, these results for AthenaPK and Parthenon‐Hydro are encouraging as this is among the largest number of logical GPUs that both applications have been run across. Additional weak‐ and strong‐scaling results for these applications and cross‐system comparisons can be found in a recent article 8 …”

“…A primary reason has been that these codes have successfully demonstrated large-scale performance portability across various leadership-class HPC systems. 8 The codes achieve this by making effective use of the Kokkos performance portability layer at scale. This is of particular interest to the OLCF due to the widespread use of Kokkos among Exascale Computing Project codes.…”

“…Parthenon 8 is a performance portable block‐structured AMR framework whose roots go back to the host‐only Athena++ 9 code. Performance portability in Parthenon is achieved through an intermediate abstraction layer that exposes a simplified interface to Kokkos 10 and is tailored to the ease of use by Parthenon (or downstream code) developers.…”