H-AMR: A New GPU-accelerated GRMHD Code for Exascale Computing with 3D Adaptive Mesh Refinement and Local Adaptive Time Stepping

Abstract

General relativistic magnetohydrodynamic (GRMHD) simulations have revolutionized our understanding of black hole accretion. Here, we present a GPU-accelerated GRMHD code H-AMR with multifaceted optimizations that, collectively, accelerate computation by 2–5 orders of magnitude for a wide range of applications. First, it introduces a spherical grid with 3D adaptive mesh refinement that operates in each of the three dimensions independently. This allows us to circumvent the Courant condition near the polar singularity, which otherwise cripples high-resolution computational performance. Second, we demonstrate that local adaptive time stepping on a logarithmic spherical-polar grid accelerates computation by a factor of ≲10 compared to traditional hierarchical time-stepping approaches. Jointly, these unique features lead to an effective speed of ∼109 zone cycles per second per node on 5400 NVIDIA V100 GPUs (i.e., 900 nodes of the OLCF Summit supercomputer). We illustrate H-AMR's computational performance by presenting the first GRMHD simulation of a tilted thin accretion disk threaded by a toroidal magnetic field around a rapidly spinning black hole. With an effective resolution of 13,440 × 4608 × 8092 cells and a total of ≲22 billion cells and ∼0.65 × 108 time steps, it is among the largest astrophysical simulations ever performed. We find that frame dragging by the black hole tears up the disk into two independently precessing subdisks. The innermost subdisk rotation axis intermittently aligns with the black hole spin, demonstrating for the first time that such long-sought alignment is possible in the absence of large-scale poloidal magnetic fields.

Radiation Transport Two-temperature GRMHD Simulations of Warped Accretion Disks

“…This problem can be solved if a smaller precessing disk tears of from a larger nonprecessing disk due to the differential frame dragging (Lense & Thirring 1918) of spacetime by a spinning BH. This process, called disk tearing, was observed in various smoothed particle hydrodynamics (SPH) simulations (e.g., Nixon et al 2012aNixon et al , 2012bNealon et al 2015; and recently confirmed by GRMHD simulations (Liska et al 2021(Liska et al , 2022b. Disk tearing, interstingly, also leads to a rapid burst of accretion, possibly explaining the large luminosities and spectral swings observed in accreting BHs (e.g., Nixon & Salvesen 2013;.…”

“…In this work we utilize the radiative version of our GPU accelerated GRMHD code H-AMR (Liska et al 2018(Liska et al , 2022b. H-AMR evolves the ideal GRMHD equations and radiation transfer equations in addition to the electron and ion entropies.…”

“…We then use beyond r  4 r g up to three layers of adaptive mesh refinement (AMR) to increase this resolution progressively in the disk from the base resolution at r ∼ 4 r g to N r × N θ × N f = 6720 × 2304 × 4096 at r ∼ 10 r g . (In addition, we use static mesh refinement; Liska et al 2022b; to reduce the fresolution to N f = 16 at the polar axis.) This is exactly half of the resolution used in Liska et al (2022b) and Musoke et al (2023).…”

“…(In addition, we use static mesh refinement; Liska et al 2022b; to reduce the fresolution to N f = 16 at the polar axis.) This is exactly half of the resolution used in Liska et al (2022b) and Musoke et al (2023). We place the radial boundaries at R in = 1.13 r g ≈ 0.84 r H and R out = 10 5 r g , which ensures that both boundaries are causally disconnected from the flow; here r a r 1 1…”

“…Thus, the results of Musoke et al (2023) suggest that disk tearing is a very promising mechanism to explain the multiwavelength variability in many accreting BHs. However, all GRMHD simulations that found tearing (Liska et al 2021(Liska et al , 2022b have relied on a cooling function (Noble et al 2009) to set the temperature and scale height of the disk artificially. In reality, the temperature of the disk is determined by complex physics involving viscous heating, radiative cooling, and the advection of energy.…”