Selecting a climate model subset to optimise key ensemble properties

Abstract

Abstract. End-users studying impacts and risks caused by human-induced climate change are often presented with large multi-model ensembles of climate projections whose composition and size are arbitrarily determined. An efficient and versatile method that finds a subset which maintains certain key properties from the full ensemble is needed, but very little work has been done in this area. Therefore, users typically make their own somewhat subjective subset choices and commonly use the equally-weighted model mean as a best estimate. However, different climate model simulations cannot necessarily be regarded 5 as independent estimates due to the presence of duplicated code and shared development history.Here, we present an efficient and flexible tool that makes better use of the ensemble as a whole by finding a subset with improved mean performance compared to the multi-model mean while at the same time maintaining the spread and addressing the problem of model interdependence. Out-of-sample skill and reliability are demonstrated using model-as-truth experiments.This approach is illustrated with one set of optimisation criteria but we also highlight the flexibility of cost functions, depend-10 ing on the focus of different users. The technique is useful for a range of applications that, for example, minimise present day bias to obtain an accurate ensemble mean, reduce dependence in ensemble spread, maximise future spread, ensure good performance of individual models in an ensemble, reduce the ensemble size while maintaining important ensemble characteristics, or optimize several of these at the same time. As in any calibration exercise, the final ensemble is sensitive to the metric, observational product and pre-processing steps used.

Philosophical Perspectives on Earth System Modeling: Truth, Adequacy, and Understanding

“…That is, they should try to identify metrics that will be most informative about the model's adequacy for that purpose, giving greater attention in to performance on those that are thought to be most relevant. The selection of relevant performance metrics will often rely on “process understanding,” especially understanding of which processes in the system strongly shape the behavior or phenomenon of interest (Baumberger et al, ; Eyring et al, ; Herger et al, ). It is important to keep in mind, however, that for many purposes, the evaluation of a model's adequacy‐for‐purpose should consider more than just performance: the model's resolution, which simplifications and idealizations it includes, how and to which data it has been tuned, and so on, can all be relevant considerations.…”

Calibrating Climate Model Ensembles for Assessing Extremes in a Changing Climate

“…Choosing one model per institute removes multiple initial condition members of the same model as well as similar or similarly calibrated models. By doing this the average model‐to‐model distances are expected to become more similar to the average model‐to‐observation distances (Herger et al, ). Indeed Figure S1a shows that for surface air temperature, the average Kolmogorov‐Smirnov (KS) test statistic between these 21 simulations and the land‐only gridded observational product CRU‐TS, v4.00 (Harris et al, ) is generally smaller than the mean model‐model KS value.…”

“…Furthermore, model‐simulated extremes may be systematically biased across various models compared to observations/reanalyses (Angélil et al, ; Bellprat & Doblas‐Reyes, ; Christensen et al, ; Donat et al, ; Wang et al, ), and therefore, taking the median or mean of the metric of interest across ensemble members can be unreliable (King & Karoly, ; King et al, ; Lewis & King, ; Perkins‐Kirkpatrick & Gibson, ). Such biases are not necessarily reduced after the poorest performing models have been removed from an ensemble; indeed, this process can reinforce model biases if metrics are not carefully chosen, since the best performing models might have common biases due to shared model development history (so‐called model interdependence; Herger et al, ).…”

“…A solution is out‐of‐sample testing using long observational records or model‐as‐truth experiments, which are common in some areas of climate science (Abramowitz & Bishop, ; Herger et al, ; Knutti et al, ; Sanderson, Wehner, et al, ) but appear to be sparse in others such as in the extremes community where they are critically needed. In this study we test one quantile‐based bias correction method (Jeon et al, ; hereinafter referred to as the Jeon method ).…”

“…Apart from testing the out‐of‐sample skill of the Jeon method, we also detail a new method to correct for biased model distribution shapes in multimodel ensembles. The technique selects the subset of climate simulations from a multimodel ensemble that reduces distribution biases (when compared to a model‐as‐truth), following the flexible approach introduced in Herger et al (). Here a modeled distribution is obtained by pooling data from a collection of climate models.…”