Nine time steps: ultra-fast statistical consistency testing of the Community Earth System Model (pyCECT v3.0)

Abstract

Abstract. The Community Earth System Model Ensemble Consistency Test (CESM-ECT) suite was developed as an alternative to requiring bitwise identical output for quality assurance. This objective test provides a statistical measurement of consistency between an accepted ensemble created by small initial temperature perturbations and a test set of CESM simulations. In this work, we extend the CESM-ECT suite with an inexpensive and robust test for ensemble consistency that is applied to Community Atmospheric Model (CAM) output after only nine model time steps. We demonstrate that adequate ensemble variability is achieved with instantaneous variable values at the ninth step, despite rapid perturbation growth and heterogeneous variable spread. We refer to this new test as the Ultra-Fast CAM Ensemble Consistency Test (UF-CAM-ECT) and demonstrate its effectiveness in practice, including its ability to detect small-scale events and its applicability to the Community Land Model (CLM). The new ultra-fast test facilitates CESM development, porting, and optimization efforts, particularly when used to complement information from the original CESM-ECT suite of tools.

Optimizing high-resolution Community Earth System Model on a heterogeneous many-core supercomputing platform

“…We use the CESM ensemble consistency test (ECT) (e.g. Baker et al, 2015;Milroy et al, 2018) to evaluate the simulation results. In particular, we use the UF-CAM-ECT test (Milroy et al, 2018) tool from CESM-ECT to determine whether three short test runs (of nine time steps in length) on the TAMU and QNLM machines are statistically indistinguishable from a large "control" ensemble (1000 members) of simulations generated on NCAR's Cheyenne machine.…”

“…Baker et al, 2015;Milroy et al, 2018) to evaluate the simulation results. In particular, we use the UF-CAM-ECT test (Milroy et al, 2018) tool from CESM-ECT to determine whether three short test runs (of nine time steps in length) on the TAMU and QNLM machines are statistically indistinguishable from a large "control" ensemble (1000 members) of simulations generated on NCAR's Cheyenne machine. The general ECT idea is that the large control ensemble, whose spread is created by round-off-level perturbations to the initial temperature, represents the natural variability in the climate model system and thus can serve as a baseline against which modified simulations can be statistically compared using principal component (PC) analysis on more than 100 physical variables including normal atmosphere state variables such as wind, temperature, moisture, etc.…”

“…Second, it is possible for CESM-ECT to legitimately fail at nine time steps due to a statistical difference at that time step and then issue a pass for the same simulation when evaluating the 1-year annual average. This conflicting result means that the difference is not significant over the longer term (an example of such a scenario is changing the type of random number generator, as detailed in Milroy et al, 2018). For the CESM-ECT tools, the nine-time-step test is always Figure 10.…”

“…Unfortunately, due to the computational cost required, this task remains future work. The CESM-ECT is a sensitive test, and it will be challenging to determine the reason for failure even with the current tools available (Milroy et al, 2019). However, in the meantime, the results of the multi-century control simulation (as discussed in Sect.…”

Making Root Cause Analysis Feasible for Large Code Bases

“…This example, RAND-MT, involves replacing the CESM default pseudorandom number generator (PRNG) with the Mersenne Twister. This experiment appears in [24] as an example that results in a UF-CAM-ECT failure. The random number generator is used to calculate distributions of cloud-related CAM variables, and this experiment is interesting because it is not a bug (in the usual sense of being incorrect) and not localized to a single line.…”