Scoring Probabilistic Forecasts: The Importance of Being Proper

Bröcker, Jochen; Smith, Leonard A.

doi:10.1175/waf966.1

Cited by 163 publications

(162 citation statements)

References 12 publications

Supporting

Mentioning

162

Contrasting

Order By: Relevance

“…If the forecaster is concerned to achieve the best possible score, then it may be in his/her interests to issue a forecast that is inconsistent with his/her beliefs about what is likely to happen. A strictly proper score is one for which the forecaster uniquely optimizes the expected score by forecasting his/her true beliefs (Bröcker and Smith, 2007;Jolliffe, 2008). In most situations a strictly proper score would be desirable so that the forecaster does not issue a misleading forecast.…”

Section: Proprietymentioning

confidence: 99%

Understanding forecast verification statistics

Mason

2008

Meteorological Applications

View full text Add to dashboard Cite

ABSTRACT:Although there are numerous reasons for performing a verification analysis, there are usually two general questions that are of interest: are the forecasts good, and can we be confident that the estimate of forecast quality is not misleading? When calculating a verification score, it is not usually obvious how the score can answer either of these questions. Some procedures for attempting to answer the questions are reviewed, with particular focus on p-values and confidence intervals. P -values are shown to be rather unhelpful in answering either question, especially when applied to probabilistic verification scores, and confidence intervals are to be preferred. However, confidence intervals cannot reveal biases in the value of a score that arises from an inadequate experimental design for testing on truly out-of-sample observations. Some specific problems with cross validation are highlighted. Finally, in the interests of increasing the insight into forecast strengths and weaknesses and in pointing towards methods for improving forecast quality, a plea is made for a more discriminating selection of verification procedures than has been adopted to date.

show abstract

Section: Proprietymentioning

confidence: 99%

Understanding forecast verification statistics

Mason

2008

Meteorological Applications

View full text Add to dashboard Cite

show abstract

“…Several different attributes of the forecast ensemble are often quantified, including model bias and ensemble spread, as well as the correspondence between forecast and outcome pairs using a variety of statistical metrics (Jolliffe and Stephenson 2003). Such metrics include deterministic skill scores, which consider the ensemble mean properties of a set of predictions, and probabilistic skill scores (Bröcker and Smith 2007) that quantify the quality of the full distribution of ensemble members relative to a reference forecast system (such as climatology, persistence or another forecast system) (Suckling and Smith 2013). Reliability measures the correspondence between the predicted probabilities and observed frequencies of a particular set of events.…”

Section: Assessing the Quality Of Climate Forecastsmentioning

confidence: 99%

Seasonal-to-Decadal Climate Forecasting

Suckling

2018

Weather &Amp; Climate Services for the Energy Industry

View full text Add to dashboard Cite

Forecasting climate over the near-term, from a season to decades ahead, has the potential to inform decision-making within the energy sector in a number of ways: from energy trading to scheduling maintenance and resource management. Recent advances in forecasting at these timescales have led to promising levels of skill in predicting the large-scale drivers of seasonal and multi-annual climate variability as well as the consequent local climate impacts of relevance for the energy sector (e.g. seasonal temperatures and wind speeds). This chapter discusses the unique aspects of near-term prediction, how it differs from the task of weather prediction and long-term climate projections, the sources of predictability on these timescales, as well as some of the current climate forecasting tools and products aiming to provide value to the energy sector.

show abstract

“…Bröcker and Smith (2006) show that only proper scores provide internally consistent evaluations of probability forecasts. The BS is not the only proper scoring rule (Gneiting and Raftery, 2004;Bröcker and 186 M. S. ROULSTON Smith, 2006), but it is probably the most widely used in the verification of operational probability forecasts. An example of an improper scoring rule is the naive linear scoring rule defined by 2 LS = (1 − f ) if the event occurs f if the event does not occur (6)…”

Section: Introductionmentioning

confidence: 95%

“…The BS discourages hedging and this is generally regarded as a highly desirable trait for a forecaster performance metric (Winkler and Murphy, 1969;Lindley, 1985;Murphy and Winkler, 1987;Wilks, 1995;Potts, 2002;Gneiting and Raftery, 2004). Bröcker and Smith (2006) show that only proper scores provide internally consistent evaluations of probability forecasts. The BS is not the only proper scoring rule (Gneiting and Raftery, 2004;Bröcker and 186 M. S. ROULSTON Smith, 2006), but it is probably the most widely used in the verification of operational probability forecasts.…”

Section: Introductionmentioning

confidence: 99%

Performance targets and the Brier score

Roulston

2007

Meteorological Applications

View full text Add to dashboard Cite

ABSTRACT:The Brier score and Brier skill score are commonly used measures for evaluating probability forecasts. An oft-cited desirable property of these measures is that they are strictly proper, meaning that forecasters wishing to optimise the expected value of these measures should report their true beliefs and not 'hedge' the forecast by reporting probabilities that differ from their true beliefs. In this article, it is argued that the propriety of the Brier score only discourages hedging if the utility of the forecaster is a linear function of the score. In particular, it is demonstrated, that if known performance targets based on the score are imposed on the forecaster then this negates the propriety of the score. Forecasters wishing to maximise the probability of achieving the target value of the Brier score or Brier skill score should hedge their forecasts rather than report their true beliefs. In contrast, performance targets drawn from a uniform distribution which are unknown to the forecaster during the evaluation period will induce the utility function of the forecaster to be a linear function of the Brier score. This preserves the propriety of the scoring rule.  Crown

show abstract

Scoring Probabilistic Forecasts: The Importance of Being Proper

Cited by 163 publications

References 12 publications

Understanding forecast verification statistics

Understanding forecast verification statistics

Seasonal-to-Decadal Climate Forecasting

Performance targets and the Brier score

Contact Info

Product

Resources

About