Coarse-grained component concurrency in Earth system modeling: parallelizing atmospheric radiative transfer in the GFDL AM3 model using the Flexible Modeling System coupling framework

Abstract

Abstract. Climate models represent a large variety of processes on a variety of timescales and space scales, a canonical example of multi-physics multi-scale modeling. Current hardware trends, such as Graphical Processing Units (GPUs) and Many Integrated Core (MIC) chips, are based on, at best, marginal increases in clock speed, coupled with vast increases in concurrency, particularly at the fine grain. Multiphysics codes face particular challenges in achieving finegrained concurrency, as different physics and dynamics components have different computational profiles, and universal solutions are hard to come by.We propose here one approach for multi-physics codes. These codes are typically structured as components interacting via software frameworks. The component structure of a typical Earth system model consists of a hierarchical and recursive tree of components, each representing a different climate process or dynamical system. This recursive structure generally encompasses a modest level of concurrency at the highest level (e.g., atmosphere and ocean on different processor sets) with serial organization underneath.We propose to extend concurrency much further by running more and more lower-and higher-level components in parallel with each other. Each component can further be parallelized on the fine grain, potentially offering a major increase in the scalability of Earth system models.We present here first results from this approach, called coarse-grained component concurrency, or CCC. Within the Geophysical Fluid Dynamics Laboratory (GFDL) Flexible Modeling System (FMS), the atmospheric radiative transfer component has been configured to run in parallel with a composite component consisting of every other atmospheric component, including the atmospheric dynamics and all other atmospheric physics components. We will explore the algorithmic challenges involved in such an approach, and present results from such simulations. Plans to achieve even greater levels of coarse-grained concurrency by extending this approach within other components, such as the ocean, will be discussed.

The GFDL Global Atmosphere and Land Model AM4.0/LM4.0: 2. Model Description, Sensitivity Studies, and Tuning Strategies

“…The radiative fluxes in AM4.0 have some sensitivity to the radiative time step, as discussed in Balaji et al (). Comparing simulations with a radiative time step of 3 h (the AM3 value) and 30 min (the physics time step at which all radiatively active species are updated), the difference in the net TOA flux is substantial, ∼3 W m −2 .…”

“…As the current simulations are based on relatively short AMIP runs, we have chosen to run the $24 SYPD configuration for best use of the available compute resources, as this allows individual simulations to complete within 1-2 days. For the same reason, we have forgone further scaling optimizations such as those achieved by the ''concurrent radiation'' configuration of AM4 (Balaji et al, 2016). This again serves to increase SYPD but at additional cost in CHSY.…”

CPMIP: measurements of real computational performance of Earth system models in CMIP6

“…This component architecture of ESMs is quite diverse (Alexander and Easterbrook, 2015), but typically most include at least two such components set up to run concurrently, in a mode we term coarse-grained concurrency (Balaji et al, 2016). This raises issues of load balance, configuring components to execute in roughly the same amount of time, so no processors sit idle.…”

“…The coupling overhead must be taken into account in an ESM performance study. Coarse-grained concurrency may be increasingly prevalent in ESM architectures in the future, because of current hardware trends (Balaji et al, 2016). The parallel component layout of some typical ESMs is shown in Fig.…”

<TT>sympl</TT> (v. 0.3.2) and <TT>climt</TT> (v. 0.11.0) – Towards a flexible framework for building model hierarchies in Python

“…Creating such an abstraction 5 is challenging due to the introduction of new architectures and computing models such as Graphics Processing Units (GPUs) and many-core CPUs (See introduction in Balaji et al, 2016). The modelling framework must facilitate storage of information about the computational resources available so that components that can take advantage of certain specialised hardware will be able to do so.…”