Probabilistic Forecasts: Scoring Rules and Their Decomposition and Diagrammatic Representation via Bregman Divergences

Abstract

Abstract:A scoring rule is a device for evaluation of forecasts that are given in terms of the probability of an event. In this article we will restrict our attention to binary forecasts. We may think of a scoring rule as a penalty attached to a forecast after the event has been observed. Thus a relatively small penalty will accrue if a high probability forecast that an event will occur is followed by occurrence of the event. On the other hand, a relatively large penalty will accrue if this forecast is followed by non-occurrence of the event.Meteorologists have been foremost in developing scoring rules for the evaluation of probabilistic forecasts. Here we use a published meteorological data set to illustrate diagrammatically the Brier score and the divergence score, and their statistical decompositions, as examples of Bregman divergences. In writing this article, we have in mind environmental scientists and modellers for whom meteorological factors are important drivers of biological, physical and chemical processes of interest. In this context, we briefly draw attention to the potential for probabilistic forecasting of the within-season component of nitrous oxide emissions from agricultural soils.

Distribution‐Based Model Evaluation and Diagnostics: Elicitability, Propriety, and Scoring Rules for Hydrograph Functionals

“…Water Resources Research Hughes and Topp (2015) to provide a diagrammatic interpretation of the Brier scoring rule and associated score divergences.…”

“…TableE1is taken fromHughes and Topp (2015) and summarizes a data set of n = 346 forecasts of 24-hr precipitation probability made by the Finnish Meteorological Institute during 2003 for the city of Tampere in southcentral Finland. The left block presents the original data, and the right block lists the data used in our case study.Forecast probabilities of rainfall p k ; k = (1, …, m) were issued using m = 11 categories.…”

“…Finally, n k /n corresponds to the relative frequency of each forecast category. We refer readers toHughes and Topp (2015) for a more detailed description of the data set. The raw precipitation data can be found at https://www.cawcr.gov.au/projects/…”

Eliciting Truthful Data From Crowdsourced Wireless Monitoring Modules in Cloud Managed Networks

“…Logarithmic scoring rule is also used to elicit the agent's beliefs in terms of subjective probabilities. However, logarithmic scoring rule attaches larger penalties than the quadratic scoring rule [28]. The logarithmic scoring rule deducts for inaccuracy by adding the natural log of the occurred event's probability from the base score [26].…”