AMReX: Block-Structured Adaptive Mesh Refinement for Multiphysics Applications

Abstract

Block-structured adaptive mesh refinement (AMR) provides the basis for the temporal and spatial discretization strategy for a number of ECP applications in the areas of accelerator design, additive manufacturing, astrophysics, combustion, cosmology, multiphase flow, and wind plant modelling. AMReX is a software framework that provides a unified infrastructure with the functionality needed for these and other AMR applications to be able to effectively and efficiently utilize machines from laptops to exascale architectures. AMR reduces the computational cost and memory footprint compared to a uniform mesh while preserving accurate descriptions of different physical processes in complex multi-physics algorithms. AMReX supports algorithms that solve systems of partial differential equations (PDEs) in simple or complex geometries, and those that use particles and/or particle-mesh operations to represent component physical processes. In this paper, we will discuss the core elements of the AMReX framework such as data containers and iterators as well as several specialized operations to meet the needs of the application projects. In addition we will highlight the strategy that the AMReX team is pursuing to achieve highly performant code across a range of acceleratorbased architectures for a variety of different applications.

What sustained multi-disciplinary research can achieve: The space weather modeling framework

“…We also developed a new PIC code, the Flexible Exascale Kinetic Simulator (FLEKS), to be used in MHD-AEPIC. FLEKS is based on the AMReX library (Zhang et al, 2019b(Zhang et al, , 2020 and it was designed for flexibility and high performance with a state- of-the-art semi-implicit PIC algorithm. Novel particle splitting and merging algorithms have been designed for FLEKS to control the number of macro-particles per cell during long MHD-AEPIC simulations.…”

In-situ assessment of device-side compute work for dynamic load balancing in a GPU-accelerated PIC code

“…Our study is designed as a general improvement in AMReX, a performance-portable, block-structured framework which provides infrastructure for mesh-based simulations, with capability to support particles [24]. For the remainder of this paper, we will use AMReX terminology, as it is consistent with our study and the naming convention is highly descriptive.…”

“…These techniques are demonstrated in the particle-in-cell (PIC) framework WarpX [21], which, as a particle-mesh simulation code, models fields via an Eulerian description and particles via a discretization in Lagrangian markers. In WarpX, multi-node parallelism is achieved through spatial domain decomposition, which is supported through the block-structured framework AMReX [24]. WarpX's particles are used to model kinetic phenomena and often cause localized spikes in memory consumption and compute demand.…”

“…The cost per GPU is taken as the sum of costs over all boxes managed by it; an efficiency of 1 indicates a perfectly balanced distribution. By default, the new distribution mapping will not be broadcast to all GPUs (line 17), but if the proposed efficiency exceeds the current efficiency by a user-selected amount (lines [18][19][20][21], the new distribution is communicated to all GPUs and updated (lines [23][24][25][26][27][28][29][30][31]. Update of the distribution mapping includes shuffling ownership of boxes as well as redistributing particles to GPUs (line 30), and when ran is the most expensive step of the load balancing routine (this is the case for simulations we present in Sec.…”

Enabling GPU Accelerated Computing in the SUNDIALS Time Integration Library

“…The packages are all built on a set of shared vector, matrix, and solver classes and are designed to be easily incorporated into existing application codes. As such, SUNDIALS has been integrated into numerous mathematical libraries and application codes including projects for block-structured adaptive mesh refinement [2], high-order finite elements [3], cosmology [4], combustion [5,6], and materials science [7], all targeting exascale computing systems.…”