Increasing the Reliability of Reliability Diagrams

Bröcker, Jochen; Smith, Leonard A.

doi:10.1175/waf993.1

Cited by 265 publications

(216 citation statements)

References 7 publications

Supporting

Mentioning

215

Contrasting

Order By: Relevance

“…Our consistency bars are then pointwise consistency bars. Also, note that the reliability diagrams we consider here are for density forecasts of continuous variables and thus somewhat different from those considered for probabilistic forecasting of binary/categorical variables (Bröcker and Smith, 2007a). The argument developed in the present article regarding the fact that serial correlation effects should be accounted for is still valid, however.…”

Section: Reliability Diagrams For Non-parametric Density Forecasts Ofmentioning

confidence: 79%

“…In practice, however, evaluation sets consisting of forecast-verification pairs are of finite (and often quite limited) size, and it is not expected that observed proportions lie exactly along the diagonal, even if the density forecasts are perfectly reliable. This issue is discussed in detail in Jolliffe and Stephenson (2003) and Bröcker and Smith (2007a), while a more general discussion on the uncertainty of verification measures can be found in Jolliffe (2007). Our contribution concerns the fact that not only sampling effects but also serial correlation in sequences of forecastverification pairs may affect the observed reliability of even perfectly reliable density forecasts of continuous variables.…”

Section: Reliability Diagrams For Non-parametric Density Forecasts Ofmentioning

confidence: 99%

“…Recently, for the case of probability forecasts of binary events, Bröcker and Smith (2007a) have explained how reliability diagrams may be misinterpreted, since even for perfectly reliable probabilistic forecasts there will always be deviations from the diagonal originating from sampling effects. They have proposed an elegant framework permitting one to easily integrate information about the impact of sampling effects directly in reliability diagrams.…”

Section: Introductionmentioning

confidence: 99%

“…While their point is highly relevant, it is argued here that it is not only counting statistics but also serial correlation in forecast-verification pairs that affects the interpretation of reliability diagrams produced from finite-size datasets. Somehow, the methodology proposed by Bröcker and Smith (2007a) is based on the fallacy that for reliable forecasts the random variables with realizations given by the probability integral transforms should be independent and identically distributed (i.i.d. ), U [0,1].…”

Section: Introductionmentioning

confidence: 99%

“…The role of serial correlation and sampling effects has already been hinted at by Pinson, et al (2007b) for the case of the verification of wind-power density forecasts, or by Hamill (2001) when considering the verification of ensemble forecasts of meteorological variables. Section 4 introduces a methodology to provide consistency bars in the spirit of Bröcker and Smith (2007a), but accounting for serial correlation effects, using an original surrogate consistency resampling method. Simulations in section 5 show how serial correlation adds to the sampling effect, and demonstrate the pitfall stemming from inference of the confidence one may have in reliability diagrams based on an i.i.d.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Reliability diagrams for non‐parametric density forecasts of continuous variables: Accounting for serial correlation

Pinson

McSharry

Madsen

2010

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

Reliability is seen as a primary requirement when verifying probabilistic forecasts, since a lack of reliability would introduce a systematic bias in subsequent decisionmaking. Reliability diagrams comprise popular and practical diagnostic tools for the reliability evaluation of density forecasts of continuous variables. Such diagrams relate to the assessment of the unconditional calibration of probabilistic forecasts. A reason for their appeal is that deviations from perfect reliability can be visually assessed based on deviations from the diagonal. Deviations from the diagonal may, however, be caused by both sampling effects and serial correlation in the forecastverification pairs. We build on a recent proposal, consisting of associating reliability diagrams with consistency bars that would reflect the deviations from the diagonal that are potentially observable even if density forecasts are perfectly reliable. Our consistency bars, however, reflect potential deviations originating from the combined effects of limited counting statistics and serial correlation in the forecastverification pairs. They are generated based on an original surrogate consistency resampling method. Its ability to provide consistency bars with a significantly better coverage against the independent and identically distributed (i.i.d.) resampling alternative is shown from simulations. Finally, a practical example of the reliability assessment of non-parametric density forecasts of short-term wind-power generation is given.

show abstract

Section: Reliability Diagrams For Non-parametric Density Forecasts Ofmentioning

confidence: 79%

Section: Reliability Diagrams For Non-parametric Density Forecasts Ofmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Reliability diagrams for non‐parametric density forecasts of continuous variables: Accounting for serial correlation

Pinson

McSharry

Madsen

2010

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

show abstract

Retracted: Reliable probabilistic forecasts from an ensemble reservoir inflow forecasting system

2014

View full text Add to dashboard Cite

This paper describes a probabilistic reservoir inflow forecasting system that explicitly attempts to sample from major sources of uncertainty in the modelling chain. Uncertainty in hydrologic forecasts arises due to errors in the hydrologic models themselves, their parameterizations, and in the initial and boundary conditions (e.g., meteorological observations or forecasts) used to drive the forecasts. The Member-to-Member (M2M) ensemble presented herein uses individual members of a numerical weather model ensemble to drive two different distributed hydrologic models, each of which is calibrated using three different objective functions. An ensemble of deterministic hydrologic states is generated by spinning up the daily simulated state using each model and parameterization. To produce probabilistic forecasts, uncertainty models are used to fit probability distribution functions (PDF) to the bias-corrected ensemble. The parameters of the distribution are estimated based on statistical properties of the ensemble and past verifying observations. The uncertainty model is able to produce reliable probability forecasts by matching the shape of the PDF to the shape of the empirical distribution of forecast errors. This shape is found to vary seasonally in the case-study watershed. We present an "intelligent" adaptation to a Probability Integral Transform (PIT)-based probability calibration scheme that relabels raw cumulative probabilities into calibrated cumulative probabili-X -4 BOURDIN ET AL.: RELIABLE PROBABILISTIC INFLOW FORECASTS

show abstract

Reliable probabilistic forecasts from an ensemble reservoir inflow forecasting system

Bourdin

Nipen

Stull

2014

Water Resources Research

View full text Add to dashboard Cite

This paper describes a probabilistic reservoir inflow forecasting system that explicitly attempts to sample from major sources of uncertainty in the modeling chain. Uncertainty in hydrologic forecasts arises due to errors in the hydrologic models themselves, their parameterizations, and in the initial and boundary conditions (e.g., meteorological observations or forecasts) used to drive the forecasts. The Member-to-Member (M2M) ensemble presented herein uses individual members of a numerical weather model ensemble to drive two different distributed hydrologic models, each of which is calibrated using three different objective functions. An ensemble of deterministic hydrologic states is generated by spinning up the daily simulated state using each model and parameterization. To produce probabilistic forecasts, uncertainty models are used to fit probability distribution functions (PDF) to the bias-corrected ensemble. The parameters of the distribution are estimated based on statistical properties of the ensemble and past verifying observations. The uncertainty model is able to produce reliable probability forecasts by matching the shape of the PDF to the shape of the empirical distribution of forecast errors. This shape is found to vary seasonally in the case-study watershed. We present an ''intelligent'' adaptation to a Probability Integral Transform (PIT)-based probability calibration scheme that relabels raw cumulative probabilities into calibrated cumulative probabilities based on recent past forecast performance. As expected, the intelligent scheme, which applies calibration corrections only when probability forecasts are deemed sufficiently unreliable, improves reliability without the inflation of ignorance exhibited in certain cases by the original PITbased scheme.

show abstract

Increasing the Reliability of Reliability Diagrams

Cited by 265 publications

References 7 publications

Reliability diagrams for non‐parametric density forecasts of continuous variables: Accounting for serial correlation

Reliability diagrams for non‐parametric density forecasts of continuous variables: Accounting for serial correlation

Retracted: Reliable probabilistic forecasts from an ensemble reservoir inflow forecasting system

Reliable probabilistic forecasts from an ensemble reservoir inflow forecasting system

Contact Info

Product

Resources

About