Verification of Categorical Probability Forecasts

Zhang, H.; Casey, Tiernan

doi:10.1175/1520-0434(2000)015<0080:vocpf>2.0.co;2

Cited by 42 publications

(25 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Both LEPS skill scores and RPSS quantify the departure between a categorical forecast and observations in the cumulative probability space (Zhang and Casey 2000). Here we used calculations for LEPS skill scores and RPSS, which are based on agreement between actual rainfall values and predicted probabilities for intervals defined by the empirical terciles of the corresponding cross-validated forecast distributions.…”

Section: A Inferential Statistical Methods To Quantify Da and Skillmentioning

confidence: 99%

“…Every empirical, descriptive skill measure, including LEPS skill scores and RPSS, needs to be complimented by some measure of uncertainty before the information can be confidently applied in decision making (Potts et al 1996;Zhang and Casey 2000;Jolliffe 2004). Beyond assessing the skill magnitude (observed skill score), it is critically important for users of forecast systems to know the probability of such skill arising by chance, in order to avoid making decisions based on artificial or perceived skill.…”

Section: A Inferential Statistical Methods To Quantify Da and Skillmentioning

confidence: 99%

“…Zhang and Casey (2000) therefore proposed the construction of "statistical distributions" using quasi-random experiments in order to assess the significance of forecast skill. They generated 95 "quasi-random ensembles" from the rainfall time series at each station by shifting the observations 1 yr ahead at a time and replacing, after each iteration, the first observation with the last.…”

Section: A Inferential Statistical Methods To Quantify Da and Skillmentioning

confidence: 99%

See 2 more Smart Citations

Inferential, Nonparametric Statistics to Assess the Quality of Probabilistic Forecast Systems

Maia¹,

Meinke²,

Lennox³

et al. 2007

Monthly Weather Review

View full text Add to dashboard Cite

Many statistical forecast systems are available to interested users. To be useful for decision making, these systems must be based on evidence of underlying mechanisms. Once causal connections between the mechanism and its statistical manifestation have been firmly established, the forecasts must also provide some quantitative evidence of "quality." However, the quality of statistical climate forecast systems (forecast quality) is an ill-defined and frequently misunderstood property. Often, providers and users of such forecast systems are unclear about what quality entails and how to measure it, leading to confusion and misinformation. A generic framework is presented that quantifies aspects of forecast quality using an inferential approach to calculate nominal significance levels ( p values), which can be obtained either by directly applying nonparametric statistical tests such as Kruskal-Wallis (KW) or Kolmogorov-Smirnov (KS) or by using Monte Carlo methods (in the case of forecast skill scores). Once converted to p values, these forecast quality measures provide a means to objectively evaluate and compare temporal and spatial patterns of forecast quality across datasets and forecast systems. The analysis demonstrates the importance of providing p values rather than adopting some arbitrarily chosen significance levels such as 0.05 or 0.01, which is still common practice. This is illustrated by applying nonparametric tests (such as KW and KS) and skill scoring methods [linear error in the probability space (LEPS) and ranked probability skill score (RPSS)] to the five-phase Southern Oscillation index classification system using historical rainfall data from Australia, South Africa, and India. The selection of quality measures is solely based on their common use and does not constitute endorsement. It is found that nonparametric statistical tests can be adequate proxies for skill measures such as LEPS or RPSS. The framework can be implemented anywhere, regardless of dataset, forecast system, or quality measure. Eventually such inferential evidence should be complemented by descriptive statistical methods in order to fully assist in operational risk management.

show abstract

Section: A Inferential Statistical Methods To Quantify Da and Skillmentioning

confidence: 99%

Section: A Inferential Statistical Methods To Quantify Da and Skillmentioning

confidence: 99%

Section: A Inferential Statistical Methods To Quantify Da and Skillmentioning

confidence: 99%

See 1 more Smart Citation

Inferential, Nonparametric Statistics to Assess the Quality of Probabilistic Forecast Systems

Maia¹,

Meinke²,

Lennox³

et al. 2007

Monthly Weather Review

View full text Add to dashboard Cite

show abstract

“…The ranked probability score (RPS) is a skill measurement that penalizes forecast errors in terms of the probability assigned to the events [59], and is based on the square error between the cumulative probabilities of forecasts and observations. This score is sensitive to distance, i.e., it includes a penalty for the forecasts that are further away of the observations.…”

Section: Forecast Verificationmentioning

confidence: 99%

Probabilistic Forecasting of Drought Events Using Markov Chain- and Bayesian Network-Based Models: A Case Study of an Andean Regulated River Basin

Avilés

Célleri

Solera

et al. 2016

Water

View full text Add to dashboard Cite

Abstract:The scarcity of water resources in mountain areas can distort normal water application patterns with among other effects, a negative impact on water supply and river ecosystems. Knowing the probability of droughts might help to optimize a priori the planning and management of the water resources in general and of the Andean watersheds in particular. This study compares Markov chain-(MC) and Bayesian network-(BN) based models in drought forecasting using a recently developed drought index with respect to their capability to characterize different drought severity states. The copula functions were used to solve the BNs and the ranked probability skill score (RPSS) to evaluate the performance of the models. Monthly rainfall and streamflow data of the Chulco River basin, located in Southern Ecuador, were used to assess the performance of both approaches. Global evaluation results revealed that the MC-based models predict better wet and dry periods, and BN-based models generate slightly more accurately forecasts of the most severe droughts. However, evaluation of monthly results reveals that, for each month of the hydrological year, either the MC-or BN-based model provides better forecasts. The presented approach could be of assistance to water managers to ensure that timely decision-making on drought response is undertaken.

show abstract

“…The statistical significance of the skill scores can be evaluated using a bootstrap resampling approach (for example, see Zhang and Casey, 2000). For the small sample sizes used (N=48), the computed skill scores have large sampling uncertainties.…”

Section: Effect Of Bias Correctionmentioning

confidence: 99%

Evaluation of bias-correction methods for ensemble streamflow volume forecasts

Hashino

Bradley

Schwartz³

2007

Hydrol. Earth Syst. Sci.

159

View full text Add to dashboard Cite

Abstract. Ensemble prediction systems are used operationally to make probabilistic streamflow forecasts for seasonal time scales. However, hydrological models used for ensemble streamflow prediction often have simulation biases that degrade forecast quality and limit the operational usefulness of the forecasts. This study evaluates three biascorrection methods for ensemble streamflow volume forecasts. All three adjust the ensemble traces using a transformation derived with simulated and observed flows from a historical simulation. The quality of probabilistic forecasts issued when using the three bias-correction methods is evaluated using a distributions-oriented verification approach. Comparisons are made of retrospective forecasts of monthly flow volumes for a north-central United States basin (Des Moines River, Iowa), issued sequentially for each month over a 48-year record. The results show that all three bias-correction methods significantly improve forecast quality by eliminating unconditional biases and enhancing the potential skill. Still, subtle differences in the attributes of the bias-corrected forecasts have important implications for their use in operational decision-making. Diagnostic verification distinguishes these attributes in a context meaningful for decision-making, providing criteria to choose among bias-correction methods with comparable skill.

show abstract

Verification of Categorical Probability Forecasts

Cited by 42 publications

References 18 publications

Inferential, Nonparametric Statistics to Assess the Quality of Probabilistic Forecast Systems

Inferential, Nonparametric Statistics to Assess the Quality of Probabilistic Forecast Systems

Probabilistic Forecasting of Drought Events Using Markov Chain- and Bayesian Network-Based Models: A Case Study of an Andean Regulated River Basin

Evaluation of bias-correction methods for ensemble streamflow volume forecasts

Contact Info

Product

Resources

About