Demonstrating the value of larger ensembles in forecasting physical systems

Abstract

A B S T R A C T Ensemble simulation propagates a collection of initial states forward in time in a Monte Carlo fashion. Depending on the fidelity of the model and the properties of the initial ensemble, the goal of ensemble simulation can range from merely quantifying variations in the sensitivity of the model all the way to providing actionable probability forecasts of the future. Whatever the goal is, success depends on the properties of the ensemble, and there is a longstanding discussion in meteorology as to the size of initial condition ensemble most appropriate for Numerical Weather Prediction. In terms of resource allocation: how is one to divide finite computing resources between model complexity, ensemble size, data assimilation and other components of the forecast system. One wishes to avoid undersampling information available from the model's dynamics, yet one also wishes to use the highest fidelity model available. Arguably, a higher fidelity model can better exploit a larger ensemble; nevertheless it is often suggested that a relatively small ensemble, say Â16 members, is sufficient and that larger ensembles are not an effective investment of resources. This claim is shown to be dubious when the goal is probabilistic forecasting, even in settings where the forecast model is informative but imperfect. Probability forecasts for a 'simple' physical system are evaluated at different lead times; ensembles of up to 256 members are considered. The pure density estimation context (where ensemble members are drawn from the same underlying distribution as the target) differs from the forecasting context, where one is given a high fidelity (but imperfect) model. In the forecasting context, the information provided by additional members depends also on the fidelity of the model, the ensemble formation scheme (data assimilation), the ensemble interpretation and the nature of the observational noise. The effect of increasing the ensemble size is quantified by its relative information content (in bits) using a proper skill score. Doubling the ensemble size is demonstrated to yield a non-trivial increase in the information content (forecast skill) for an ensemble with well over 16 members; this result stands in forecasting a mathematical system and a physical system. Indeed, even at the largest ensemble sizes considered (128 and 256), there are lead times where the forecast information is still increasing with ensemble size. Ultimately, model error will limit the value of ever larger ensembles. No support is found, however, for limiting design studies to the sizes commonly found in seasonal and climate studies. It is suggested that ensemble size be considered more explicitly in future design studies of forecast systems on all time scales.

Ensemble size: How suboptimal is less than infinity?

“…To apply the logarithmic score, the pdf has been modelled by a Gaussian here, so that the score is equivalent to the Dawid–Sebastiani score, which has been shown to decrease asymptotically with ensemble size m as m −1 . Machete and Smith () examined a more general situation, in which a kernel density estimate allows different shapes of pdfs. It is an open question how such an approach will change the asymptotic convergence properties of the logarithmic score.…”

“…For an eight‐member ensemble, the DSS is about 0.2 nats larger than for a 50‐member ensemble. Machete and Smith () define a competitive advantage that depends on the difference in logarithmic scores as when the logarithmic score LS is computed with the natural logarithm. The competitive advantage of using 50 instead of 8 members turns out to be about 30%.…”

“…Machete and Smith () gave fresh impetus to the debate about the appropriate ensemble size for probabilistic forecasting. They considered ensemble forecasts for the Moore–Spiegel system in a perfect model scenario and ensemble forecasts for an electronic circuit that is a physical realization of the Moore–Spiegel system, that is, in an imperfect model scenario.…”

Can an ensemble climate simulation be used to separate climate change signals from internal unforced variability?

“…Integrations have been performed between 1850 and 2005 with the aim of assessing the value of an ensemble experiment in determining to what extent a climate change signal can be separated from internal chaotic variability, but we do not address climate change directly by using future projections as in some studies. The model used in this study represents the observed internal variability sufficiently well and it can therefore be assumed that an ensemble of 100 members will be sufficient to separate the forced mode from the internal variability (Machete and Smith 2016). The distribution of trends in temperature and precipitation are compared with the observed trends and those from previous studies.…”

Probabilistic recalibration of forecasts

“…The circuit was first introduced in [50] and later discussed in more detail in [51]. References [51] and [52] discuss different aspects of its predictability properties. The circuit is designed to produce output voltages that mimic the Moore-Spiegel [53] three-dimensional system of ordinary differential equations:…”