Effective Scoring Rules for Probabilistic Forecasts

Friedman, Daniel

doi:10.1287/mnsc.29.4.447

Cited by 52 publications

(47 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A unifying perspective on these two families of rules, which might help to provide some guidance concerning appropriate values of , has hitherto been lacking. Friedman (1983) attempted to identify scoring rules with metrics (rather than divergences) on the probability space, but most metrics turn out not to have associated scoring rules, and vice versa, as shown by Nau (1985). More recently, Selten (1998) has discussed the implications of different values of in the power scoring rule, arguing against the logarithmic rule ( = 1) because of its hypersensitivity to the estimation of small probabilities and in favor of the quadratic rule ( = 2) because the latter uniquely satisfies a certain axiom of "neutrality," namely, that the expected loss for reporting r when the true distribution is p is the same as the expected loss for reporting p when the true distribution is r, i.e., S p p − S r p = S r r − S p r .…”

Section: Weighted Scoring Rulesmentioning

confidence: 99%

Scoring Rules, Generalized Entropy, and Utility Maximization

Jose

Nau

Winkler

2008

Operations Research

View full text Add to dashboard Cite

Information measures arise in many disciplines, including forecasting (where scoring rules are used to provide incentives for probability estimation), signal processing (where information gain is measured in physical units of relative entropy), decision analysis (where new information can lead to improved decisions), and finance (where investors optimize portfolios based on their private information and risk preferences). In this paper, we generalize the two most commonly used parametric families of scoring rules and demonstrate their relation to well-known generalized entropies and utility functions, shedding new light on the characteristics of alternative scoring rules as well as duality relationships between utility maximization and entropy minimization. In particular, we show that weighted forms of the pseudospherical and power scoring rules correspond exactly to measures of relative entropy (divergence) with convenient properties, and they also correspond exactly to the solutions of expected utility maximization problems in which a risk-averse decision maker whose utility function belongs to the linear-risk-tolerance family interacts with a risk-neutral betting opponent or a complete market for contingent claims in either a one-period or a two-period setting. When the market is incomplete, the corresponding problems of maximizing linear-risk-tolerance utility with the risk-tolerance coefficient are the duals of the problems of minimizing the pseudospherical or power divergence of order between the decision maker's subjective probability distribution and the set of risk-neutral distributions that support asset prices.

show abstract

Section: Weighted Scoring Rulesmentioning

confidence: 99%

Scoring Rules, Generalized Entropy, and Utility Maximization

Jose

Nau

Winkler

2008

Operations Research

View full text Add to dashboard Cite

show abstract

“…An effective score is one which monotonically improves as the distance (however it is measured) between the forecast and the observation decreases (Friedman, 1983;Nau, 1985). The most often quoted example of a score that is ineffective is the original version of the LEPS score, which for large numbers of categories could score a forecast with an extremely large error less severely than one with only a large error (Potts et al, 1996).…”

Section: Effectivenessmentioning

confidence: 99%

Understanding forecast verification statistics

Mason

2008

Meteorological Applications

View full text Add to dashboard Cite

ABSTRACT:Although there are numerous reasons for performing a verification analysis, there are usually two general questions that are of interest: are the forecasts good, and can we be confident that the estimate of forecast quality is not misleading? When calculating a verification score, it is not usually obvious how the score can answer either of these questions. Some procedures for attempting to answer the questions are reviewed, with particular focus on p-values and confidence intervals. P -values are shown to be rather unhelpful in answering either question, especially when applied to probabilistic verification scores, and confidence intervals are to be preferred. However, confidence intervals cannot reveal biases in the value of a score that arises from an inadequate experimental design for testing on truly out-of-sample observations. Some specific problems with cross validation are highlighted. Finally, in the interests of increasing the insight into forecast strengths and weaknesses and in pointing towards methods for improving forecast quality, a plea is made for a more discriminating selection of verification procedures than has been adopted to date.

show abstract

“…Each score in turn is then illustrated in the context of decadal forecasts of global mean temperature. Section 2 discusses several measures of forecast system performance, including the logarithmic score (Ignorance) (Good 1952;Roulston and Smith 2002), the Continuous Ranked Probability Score (CRPS) (Epstein 1969;Gneiting and Raftery 2007) and the Proper Linear score (PL) (Friedman 1983). General considerations for selecting a preferred score are discussed; CRPS is demonstrated capable of misleading behaviour.…”

Section: Introductionmentioning

confidence: 99%

Towards improving the framework for probabilistic forecast evaluation

et al. 2015

View full text Add to dashboard Cite

The evaluation of forecast performance plays a central role both in the interpretation and use of forecast systems and in their development. Different evaluation measures (scores) are available, often quantifying different characteristics of forecast performance. The properties of several proper scores for probabilistic forecast evaluation are contrasted and then used to interpret decadal probability hindcasts of global mean temperature. The Continuous Ranked Probability Score (CRPS), Proper Linear (PL) score, and IJ Good's logarithmic score (also referred to as Ignorance) are compared; although information from all three may be useful, the logarithmic score has an immediate interpretation and is not insensitive to forecast busts. Neither CRPS nor PL is local; this is shown to produce counter intuitive evaluations by CRPS. Benchmark forecasts from empirical models like Dynamic Climatology place the scores in context. Comparing scores for forecast systems based on physical models (in this case HadCM3, from the CMIP5 decadal archive) against such benchmarks is more informative than internal comparison systems based on similar physical simulation models with each other. It is shown that a forecast system based on HadCM3 out performs Dynamic Climatology in decadal global mean temperature hindcasts; Dynamic Climatology previously outperformed a forecast system based upon HadGEM2 and reasons for these results are suggested. Forecasts of aggregate data (5-year means of global mean temperature) are, of course, narrower than forecasts of annual averages due to the suppression of variance; while the average "distance" between the forecasts and a target may be expected to decrease, little if any discernible improvement in probabilistic skill is achieved.

show abstract

Effective Scoring Rules for Probabilistic Forecasts

Cited by 52 publications

References 8 publications

Scoring Rules, Generalized Entropy, and Utility Maximization

Scoring Rules, Generalized Entropy, and Utility Maximization

Understanding forecast verification statistics

Towards improving the framework for probabilistic forecast evaluation

Contact Info

Product

Resources

About