Detecting Improvements in Forecast Correlation Skill: Statistical Testing and Power Analysis

Abstract

The skill of weather and climate forecast systems is often assessed by calculating the correlation coefficient between past forecasts and their verifying observations. Improvements in forecast skill can thus be quantified by correlation differences. The uncertainty in the correlation difference needs to be assessed to judge whether the observed difference constitutes a genuine improvement, or is compatible with random sampling variations. A widely used statistical test for correlation difference is known to be unsuitable, because it assumes that the competing forecasting systems are independent. In this paper, appropriate statistical methods are reviewed to assess correlation differences when the competing forecasting systems are strongly correlated with one another. The methods are used to compare correlation skill between seasonal temperature forecasts that differ in initialization scheme and model resolution. A simple power analysis framework is proposed to estimate the probability of correctly detecting skill improvements, and to determine the minimum number of samples required to reliably detect improvements. The proposed statistical test has a higher power of detecting improvements than the traditional test. The main examples suggest that sample sizes of climate hindcasts should be increased to about 40 years to ensure sufficiently high power. It is found that seasonal temperature forecasts are significantly improved by using realistic land surface initial conditions.

Multi-year prediction of European summer drought conditions for the agricultural sector

“…The former measure is the absolute difference in correlation values of the INIT and NoINIT hindcasts against observation. It is the most commonly used skill measure to assess the contribution of the internally generated climate variability component to the multiannual predictive skill (Siegert et al 2017). However, it has been pointed out by some studies (Siegert et al 2017, Smith et al 2019 that using correlation difference for two highly correlated hindcasts underestimates the impact of initialization.…”

“…It is the most commonly used skill measure to assess the contribution of the internally generated climate variability component to the multiannual predictive skill (Siegert et al 2017). However, it has been pointed out by some studies (Siegert et al 2017, Smith et al 2019 that using correlation difference for two highly correlated hindcasts underestimates the impact of initialization. Therefore, in order to determine whether initialization adds any additional information compared to the uninitialized forecast in the areas where both INIT and NoINIT return high correlation values, we apply the residual correlation methodology recently suggested by Smith et al (2019).…”

Benefits of Increasing the Model Resolution for the Seasonal Forecast Quality in EC-Earth

“…We use the Student distribution with N degrees of freedom to estimate the significance level of correlation, N being the effective number of independent data calculated following the method of von Storch and Zwiers (2001). The significance of the difference between two correlations is estimated using the methodology of Siegert et al (2016), which takes into account the dependence from sharing the same observations in both correlation coefficients. This method to assess the significance of the difference of two correlations also takes into account the independent number of data, which is necessary given the serial correlation typical of the time series considered.…”

Forecasting the climate response to volcanic eruptions: prediction skill related to stratospheric aerosol forcing

“…To test a forecast system at the seasonal and/or decadal timescales, an ensemble of several tens of predictions is commonly required to ensure enough confidence in the evaluation of the skill (Siegert et al 2017). Hence, it is statistically not possible to test a forecast system by considering only the three predictions initialised the years of the last three major eruptions.…”

“…The correlation difference between the hindcast including the volcanic forcing and the other experiments is significant at 95% when an inverted triangle is superimposed to the curve. As recommended in (Siegert et al 2017), the correlation between the different experiments is taken into account to compute the statistical significance of the skill differences correctly. These differences are never significant in (a).…”