Kullback–Leibler Divergence as a Forecast Skill Score with Classic Reliability–Resolution–Uncertainty Decomposition

Weijs, Steven; Nooijen, Ronald van; Giesen, Nick van de

doi:10.1175/2010mwr3229.1

Cited by 83 publications

(95 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Verifying the quality of forecasts expressed in a probabilistic form requires specific graphical or numerical tools (Jolliffe and Stephenson 2011), among them some numerical measures of performance such as the Brier score (Brier 1950), the Kullback-Leibler divergence (Weijs et al 2010) and many others (Winkler et al 1996;Gneiting and Raftery 2007). When the probabilistic forecast is a cumulative distribution function (CDF) and the observation is a scalar, the continuous ranked probability score (CRPS) is often used as a quantitative measure of performance.…”

Section: Introductionmentioning

confidence: 99%

Estimation of the Continuous Ranked Probability Score with Limited Information and Applications to Ensemble Weather Forecasts

Zamo

Naveau

2017

Math Geosci

View full text Add to dashboard Cite

The continuous ranked probability score (CRPS) is a much used measure of performance for probabilistic forecasts of a scalar observation. It is a quadratic measure of the difference between the forecast cumulative distribution function (CDF) and the empirical CDF of the observation. Analytic formulations of the CRPS can be derived for most classical parametric distributions, and be used to assess the efficiency of different CRPS estimators. When the true forecast CDF is not fully known, but represented as an ensemble of values, the CRPS is estimated with some error. Thus, using the CRPS to compare parametric probabilistic forecasts with ensemble forecasts may be misleading due to the unknown error of the estimated CRPS for the ensemble. With simulated data, the impact of the type of the verified ensemble (a random sample or a set of quantiles) on the CRPS estimation is studied. Based on these simulations, recommendations are issued to choose the most accurate CRPS estimator according to the type of ensemble. The interest of these recommendations is illustrated with real ensemble weather forecasts. Also, relationships between several estimators of the CRPS are demonstrated and used to explain the differences of accuracy between the estimators.

show abstract

Section: Introductionmentioning

confidence: 99%

Estimation of the Continuous Ranked Probability Score with Limited Information and Applications to Ensemble Weather Forecasts

Zamo

Naveau

2017

Math Geosci

View full text Add to dashboard Cite

show abstract

“…Using Bregman divergences [18,19], our calculations lead to identical numerical results to those outlined above, in terms of the scores and their decompositions. What we gain by the analysis presented here is a set of diagrams which usefully complement those used by Weijs et al [11,12] to illustrate the statistical decomposition both of the Brier score and the divergence score. This is possible because of the availability of a simple diagrammatic format for the illustration of Bregman divergences (e.g., [19,20]).…”

Section: Forecast Evaluation Via Bregman Divergencesmentioning

confidence: 97%

“…Weijs et al [11,12] provide informative background on the provenance of the divergence score, and a detailed analysis of its derivation. We refer interested readers this work, and present here only enough details to illustrate a template calculation of the score and its reliability-resolution-uncertainty decomposition.…”

Section: The Divergence Score and Its Decompositionmentioning

confidence: 99%

“…Murphy [9] provided a statistical decomposition of the Brier score into three components, which he termed uncertainty, reliability and resolution (see also [10]). Weijs et al [11,12] provided a further analysis of the logarithmic score, resulting in the divergence score and its statistical decomposition into the equivalent three components. The cited articles discuss uncertainty, reliability and resolution in detail.…”

Section: Introductionmentioning

confidence: 99%

“…One way of looking at the present article is as a complement to recent analytical innovations in forecast evaluation [11,12]. Using Bregman divergences, we provide a new calculation template for analysis of the Brier score and the divergence score, and new explanatory diagrams.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Probabilistic Forecasts: Scoring Rules and Their Decomposition and Diagrammatic Representation via Bregman Divergences

Hughes

Topp

2015

Entropy

View full text Add to dashboard Cite

Abstract:A scoring rule is a device for evaluation of forecasts that are given in terms of the probability of an event. In this article we will restrict our attention to binary forecasts. We may think of a scoring rule as a penalty attached to a forecast after the event has been observed. Thus a relatively small penalty will accrue if a high probability forecast that an event will occur is followed by occurrence of the event. On the other hand, a relatively large penalty will accrue if this forecast is followed by non-occurrence of the event.Meteorologists have been foremost in developing scoring rules for the evaluation of probabilistic forecasts. Here we use a published meteorological data set to illustrate diagrammatically the Brier score and the divergence score, and their statistical decompositions, as examples of Bregman divergences. In writing this article, we have in mind environmental scientists and modellers for whom meteorological factors are important drivers of biological, physical and chemical processes of interest. In this context, we briefly draw attention to the potential for probabilistic forecasting of the within-season component of nitrous oxide emissions from agricultural soils.

show abstract

Estimating information entropy for hydrological data: One‐dimensional case

Gong

Yang

Gupta

et al. 2014

Water Resources Research

View full text Add to dashboard Cite

There has been a recent resurgence of interest in the application of Information Theory to problems of system identification in the Earth and Environmental Sciences. While the concept of entropy has found increased application, little attention has yet been given to the practical problems of estimating entropy when dealing with the unique characteristics of two commonly used kinds of hydrologic data: rainfall and runoff. In this paper, we discuss four important issues of practical relevance that can bias the computation of entropy if not properly handled. The first (zero effect) arises when precipitation and ephemeral streamflow data must be viewed as arising from a discrete-continuous hybrid distribution due to the occurrence of many zero values (e.g., days with no rain/no runoff). Second, in the widely used bin-counting method for estimation of PDF's, significant error can be introduced if the bin width is not carefully selected. The third (measurement effect) arises due to the fact that continuously varying hydrologic variables can typically only be observed discretely to some degree of precision. The Fourth (skewness effect) arises when the distribution of a variable is significantly skewed. Here we present an approach that can deal with all four of these issues, and test them with artificially generated and real hydrological data. The results indicate that the method is accurate and robust.

show abstract

Kullback–Leibler Divergence as a Forecast Skill Score with Classic Reliability–Resolution–Uncertainty Decomposition

Cited by 83 publications

References 16 publications

Estimation of the Continuous Ranked Probability Score with Limited Information and Applications to Ensemble Weather Forecasts

Estimation of the Continuous Ranked Probability Score with Limited Information and Applications to Ensemble Weather Forecasts

Probabilistic Forecasts: Scoring Rules and Their Decomposition and Diagrammatic Representation via Bregman Divergences

Estimating information entropy for hydrological data: One‐dimensional case

Contact Info

Product

Resources

About