A Generic Forecast Verification Framework for Administrative Purposes

Mason, Simon J.; Weigel, Andreas

doi:10.1175/2008mwr2553.1

Cited by 72 publications

(66 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several scores are available for the evaluation of probabilistic forecasts (Bröcker and Smith 2007;Gneiting and Raftery 2007;Mason and Weigel 2009;Jolliffe and Stephenson 2012); each quantifies different attributes of the forecast. While the importance of using proper scores is well recognised (Bröcker and Smith 2007;Fricker et al 2013), researchers often face requests to present results under a variety of scores.…”

Section: Measuring Forecast Performancementioning

confidence: 99%

Towards improving the framework for probabilistic forecast evaluation

et al. 2015

View full text Add to dashboard Cite

The evaluation of forecast performance plays a central role both in the interpretation and use of forecast systems and in their development. Different evaluation measures (scores) are available, often quantifying different characteristics of forecast performance. The properties of several proper scores for probabilistic forecast evaluation are contrasted and then used to interpret decadal probability hindcasts of global mean temperature. The Continuous Ranked Probability Score (CRPS), Proper Linear (PL) score, and IJ Good's logarithmic score (also referred to as Ignorance) are compared; although information from all three may be useful, the logarithmic score has an immediate interpretation and is not insensitive to forecast busts. Neither CRPS nor PL is local; this is shown to produce counter intuitive evaluations by CRPS. Benchmark forecasts from empirical models like Dynamic Climatology place the scores in context. Comparing scores for forecast systems based on physical models (in this case HadCM3, from the CMIP5 decadal archive) against such benchmarks is more informative than internal comparison systems based on similar physical simulation models with each other. It is shown that a forecast system based on HadCM3 out performs Dynamic Climatology in decadal global mean temperature hindcasts; Dynamic Climatology previously outperformed a forecast system based upon HadGEM2 and reasons for these results are suggested. Forecasts of aggregate data (5-year means of global mean temperature) are, of course, narrower than forecasts of annual averages due to the suppression of variance; while the average "distance" between the forecasts and a target may be expected to decrease, little if any discernible improvement in probabilistic skill is achieved.

show abstract

Section: Measuring Forecast Performancementioning

confidence: 99%

Towards improving the framework for probabilistic forecast evaluation

et al. 2015

View full text Add to dashboard Cite

show abstract

“…Another score, designed to give an intuitive measure of the prediction system performance, is the generalized discrimination score, or two alternatives forced choice score (2AFC) (Mason and Weigel, 2009;Weigel and Mason, 2011). The score is based on pairs of observations and forecasts.…”

Section: Verification Scoresmentioning

confidence: 99%

“…An appealing property of the score is its applicability for ensemble, probabilistic, dichotomous, polychotomous or continuous forecasts and corresponding observations. Here, as an illustration, we give the formulation for probabilistic forecast; for all other possible combinations of forecast and observation type, referred to Mason and Weigel (2009) and Weigel and Mason (2011). Let n 1 be the number of observed events i, n 0 the number of nonevents j .…”

Section: Verification Scoresmentioning

confidence: 99%

Monthly hydrometeorological ensemble prediction of streamflow droughts and corresponding drought indices

Fundel

Jörg-Hess

Zappa

2013

Hydrol. Earth Syst. Sci.

View full text Add to dashboard Cite

Abstract. Streamflow droughts, characterized by low runoff as consequence of a drought event, affect numerous aspects of life. Economic sectors that are impacted by low streamflow are, e.g., power production, agriculture, tourism, water quality management and shipping. Those sectors could potentially benefit from forecasts of streamflow drought events, even of short events on the monthly time scales or below. Numerical hydrometeorological models have increasingly been used to forecast low streamflow and have become the focus of recent research. Here, we consider daily ensemble runoff forecasts for the river Thur, which has its source in the Swiss Alps. We focus on the evaluation of low streamflow and of the derived indices as duration, severity and magnitude, characterizing streamflow droughts up to a lead time of one month.The ECMWF VarEPS 5-member ensemble reforecast, which covers 18 yr, is used as forcing for the hydrological model PREVAH. A thorough verification reveals that, compared to probabilistic peak-flow forecasts, which show skill up to a lead time of two weeks, forecasts of streamflow droughts are skilful over the entire forecast range of one month. For forecasts at the lower end of the runoff regime, the quality of the initial state seems to be crucial to achieve a good forecast quality in the longer range. It is shown that the states used in this study to initialize forecasts satisfy this requirement. The produced forecasts of streamflow drought indices, derived from the ensemble forecasts, could be beneficially included in a decision-making process. This is valid for probabilistic forecasts of streamflow drought events falling below a daily varying threshold, based on a quantile derived from a runoff climatology. Although the forecasts have a tendency to overpredict streamflow droughts, it is shown that the relative economic value of the ensemble forecasts reaches up to 60 %, in case a forecast user is able to take preventive action based on the forecast.

show abstract

“…This study does not address the forecast value for a specific application of interest to a specific end-user. It assesses the overall forecast performance for the purpose of strategic planning (the administrative verification forecast; Mason and Weigel, 2009;Joliffe and Stephenson, 2003;Murphy, 1993;Brier and Allen, 1951). In particular: (a) cumulative statistics for the bias and root mean square errors (RMSE) are employed to assess the forecast error out to four days, and; (b) the skill of the RAMS forecast is compared by means of the Anomaly Correlation Coefficient (ACC) against those related to the persistence forecast, which is supposed to have good performance in summer because the Central Mediterranean is characterized by fair and stable weather during this season.…”

mentioning

confidence: 99%

Verification of surface minimum, mean, and maximum temperature forecasts in Calabria for summer 2008

Federico

2011

Nat. Hazards Earth Syst. Sci.

View full text Add to dashboard Cite

Abstract. Since 2005, one-hour temperature forecasts for the Calabria region (southern Italy), modelled by the Regional Atmospheric Modeling System (RAMS), have been issued by CRATI/ISAC-CNR (Consortium for Research and Application of Innovative Technologies/Institute for Atmospheric and Climate Sciences of the National Research Council) and are available online at meteo.crati.it/previsioni.html (every six hours). Beginning in June 2008, the horizontal resolution was enhanced to 2.5 km. In the present paper, forecast skill and accuracy are evaluated out to four days for the 2008 summer season (from 6 June to 30 September, 112 runs). For this purpose, gridded high horizontal resolution forecasts of minimum, mean, and maximum temperatures are evaluated against gridded analyses at the same horizontal resolution (2.5 km).Gridded analysis is based on Optimal Interpolation (OI) and uses the RAMS first-day temperature forecast as the background field. Observations from 87 thermometers are used in the analysis system. The analysis error is introduced to quantify the effect of using the RAMS first-day forecast as the background field in the OI analyses and to define the forecast error unambiguously, while spatial interpolation (SI) analysis is considered to quantify the statistics' sensitivity to the verifying analysis and to show the quality of the OI analyses for different background fields.Two case studies, the first one with a low (less than the 10th percentile) root mean square error (RMSE) in the OI analysis, the second with the largest RMSE of the whole period in the OI analysis, are discussed to show the forecast performance under two different conditions. Cumulative statistics are used to quantify forecast errors out to four days. Results show that maximum temperature has the largest RMSE, while minimum and mean temperature errors are similar. ForCorrespondence to: S. Federico (s.federico@isac.cnr.it) the period considered, the OI analysis RMSEs for minimum, mean, and maximum temperatures vary from 1.8, 1.6, and 2.0 • C, respectively, for the first-day forecast, to 2.0, 1.9, and 2.6 • C, respectively, for the fourth-day forecast.Cumulative statistics are computed using both SI and OI analysis as reference. Although SI statistics likely overestimate the forecast error because they ignore the observational error, the study shows that the difference between OI and SI statistics is less than the analysis error.The forecast skill is compared with that of the persistence forecast. The Anomaly Correlation Coefficient (ACC) shows that the model forecast is useful for all days and parameters considered here, and it is able to capture day-to-day weather variability. The model forecast issued for the fourth day is still better than the first-day forecast of a 24-h persistence forecast, at least for mean and maximum temperature. The impact of using the RAMS first-day forecast as the background field in the OI analysis is quantified by comparing statistics computed with OI and SI analyses. Minimum temperature is more sensiti...

show abstract

A Generic Forecast Verification Framework for Administrative Purposes

Cited by 72 publications

References 21 publications

Towards improving the framework for probabilistic forecast evaluation

Towards improving the framework for probabilistic forecast evaluation

Monthly hydrometeorological ensemble prediction of streamflow droughts and corresponding drought indices

Verification of surface minimum, mean, and maximum temperature forecasts in Calabria for summer 2008

Contact Info

Product

Resources

About