Enabling high-performance cloud computing for Earth science modeling on over a thousand cores: application to the GEOS-Chem atmospheric chemistry model

Abstract

Cloud computing platforms can facilitate the use of Earth science models by providing immediate access to fully configured software, massive computing power, and large input datasets. However, slow inter-node communication performance has previously discouraged the use of cloud platforms for massively parallel simulations. Here we show that recent advances in the network performance on the Amazon Web Services (AWS) cloud enable efficient model simulations with over a thousand cores. The choices of Message Passing Interface (MPI) library configuration and inter-node communication protocol are critical to this success. Application to the GEOS-Chem model of atmospheric chemistry at global 50 km horizontal resolution shows efficient scaling up to at least 1152 cores, with performance and cost comparable to the NASA Pleiades supercomputing cluster.

CHEEREIO 1.0: a versatile and user-friendly ensemble-based chemical data assimilation and emissions inversion platform for the GEOS-Chem chemical transport model

“…Further improvements to the LETKF parallelization routine, in particular methods to share memory resources within Python, can also be applied to reduce I/O overhead, reduce memory use, and improve assimilation wall time. CHEEREIO can be ported on the cloud, taking advantage of GEOS-Chem and satellite data already hosted there [Zhuang et al, 2019[Zhuang et al, , 2020Varon et al, 2022], thus bringing compute capacity to big data rather than requiring cumbersome data downloads. Cloud implementation would facilitate the development of nearreal-time chemical data assimilation products for emissions monitoring and air quality forecasts.…”