Flip‐Flop Index: Quantifying revision stability for fixed‐event forecasts

Abstract

The degree to which a forecast changes from one issue time to the next is an interesting aspect of a forecast system. Weather forecasters report that they are reluctant to change a forecast if they judge there is a risk of it being changed back again. They believe such instability detracts from the message being delivered and are reluctant to use automated guidance which they perceive as having lack of stability. A Flip‐Flop Index was developed to quantify this characteristic of revisions of fixed‐event forecasts. The index retains physically meaningful units, has a simple definition and does not penalize a sequence of forecasts that show a trend, which is important when assessing forecasts where a trend can be interpreted as a forecast becoming more confident with a shorter lead time. The Flip‐Flop Index was used to compare the stability of sequences of automated guidance with the official Australian Bureau of Meteorology forecasts, which are prepared manually. The results show that the forecasts for chance of rain from the automated guidance are often more stable than the official, manual forecasts. However, the official forecasts for maximum temperature are more stable than those based on automated guidance. The Flip‐Flop Index is independent of observations and does not measure skill, but it can play a complementary role in characterizing and evaluating a forecasting system.

Verification: Assessment of Calibration and Accuracy

“…To this end, we look at the convergence index proposed by and compare it against the raw ensemble and the EMOS forecasts for surface temperature. Other, similar tools include the flip-flop index (Griffiths et al, 2018 ), inconsistency index (Zsoter, Buizza, and Richardson, 2009 ), forecast convergence score (Ruth et al, 2009 ) and divergence index (Richardson, Cloke, Seasonal temperature forecasts and Pappenberger, 2020 ). We refer to a number of predictions for the same validity time and location as a forecast sequence.…”

Comparison of Single-Valued Forecasts in a User-Oriented Framework

“…Decisions made in the real world will not necessarily be well represented by a simple costloss proposition or a single decision threshold, and various more complex approaches have been suggested (Shorr 1966;Matte et al 2017;Roulston and Smith 2004). There are other properties of the forecast that may affect user decision-making, for instance forecast stability (Griffiths et al 2019), which is not considered in this framework. We have already noted limitations with the linear error model.…”

Streamlining the graphical forecast process

“…The picture that emerged was that the GOCF-based FirstCutForecast outputs were by and large of comparable quality to the officially produced forecasts. It was also shown through use of a 'flip-flop index' (Griffiths et al 2019) that forecasts based on consensus guidance were, in many cases, more stable than the official forecasts.…”