Scalability and some optimization of the Finite-volumE Sea ice-Ocean Model, Version 2.0 (FESOM2)

Abstract

Abstract. A study of the scalability of the Finite-volumE Sea ice-Ocean circulation Model, Version 2.0 (FESOM2), the first mature global model of its kind formulated on unstructured meshes, is presented. This study includes an analysis of main computational kernels with a special focus on bottlenecks in parallel scalability. Several model enhancements, improving this scalability for large numbers of processes, are described and tested. Model grids at different resolutions are used on four HPC systems with differing computation and communication hardware to demonstrate model's scalability and throughput. Furthermore, strategies for improvements in parallel performance are presented and assessed. We show that in terms of throughput FESOM2.0 is on par with the state-of-the-art structured ocean models and in realistic eddy resolving configuration (1/10° resolution) can produce about 16 years per day on 14 000 cores. This suggests that unstructured-mesh models are becoming extremely competitive tools in high-resolution climate modelling. It is shown that main bottlenecks of FESOM parallel scalability are the two-dimensional components of the model, namely the computations of external (barotropic) mode and the sea-ice model. It is argued that these bottlenecks are shared with other general ocean circulation models.

Evaluation of FESOM2.0 Coupled to ECHAM6.3: Preindustrial and HighResMIP Simulations

“…FESOM was run using 288 cores and 1,152 cores for LR and HR meshes, respectively. It is worth mentioning that FESOM2.0 shows almost linear scalability up to ~300 surface grid points per core (Koldunov et al, ), implying that the throughputs of the ocean component presented here could easily be increased.…”

“…While FESOM1.4 has been used in various applications, faster numerical solutions operating on unstructured meshes have been sought for, leading to a new dynamical core for FESOM (Danilov et al, ; Scholz et al, ). FESOM version 2.0 provides up to 3 times speedup compared to its predecessor version 1.4, ensuring a throughput similar to that of regular‐mesh models, while promising larger mesh flexibility and good scalability characteristics (Koldunov et al, ). The dynamical core of FESOM2.0 is based on finite volume discretization compared to the finite element discretization used for FESOM1.4; the abbreviation FESOM now reads as the Finite‐volumE Sea ice–Ocean Model.…”

Abrupt Climate and Weather Changes Across Time Scales

“…Here, traditional high‐resolution modeling approaches, involving uniform meshes, would result in prohibitive computational costs. To overcome this drawback, one can apply a coupled climate‐radiocarbon circulation model using innovative and scalable numerical schemes on supercomputing facilities (e.g., Koldunov et al, 2019). This issue deserves further efforts combining simulations and reconstructions, all the more, as uncertainties in reconstructions are large for times prior to about 30 kyr before present, where the marine reservoir variability is poorly constrained through reconstructions.…”

The Atlantic Meridional Overturning Circulation in High‐Resolution Models

“…A major bottleneck that currently applies to ocean/ocean‐atmosphere models is the limit in scalability of the model performance as the number of processors used is increased. When the number of cores is increased to O(10000‐20000), performance starts to plateau and increasing the core number even further is detrimental to model performance (e.g., Kiss et al, 2020; Koldunov et al, ). Unless there is a step change in processor technology (such as quantum computing, e.g., Arute et al, ; Steane, 1998) in the near future, one can assume that the next generation(s) of HPCs will consist of massively parallel machines with millions of cores becoming commonplace.…”

“…As an example, the Energy Exascale Earth System Model (E3SM) project (e3sm.org), funded by the U.S. Department of Energy, is developing an Earth system model based on highly scalable Earth system components (with MPAS‐Ocean being its variable resolution ocean component). Other examples include IMMERSE (https://immerse-ocean.eu), FESOM (unstructured mesh, Wang et al, ; Koldunov et al, ), ICON‐ESM (e.g., Korn, ), MPAS‐Ocean (Ringler et al, 2013), and iHESP (https://ihesp.tamu.edu). These efforts are explicitly targeted for highly resolved simulations on the next generation exascale machines.…”