Comparison of non-homogeneous regression models for probabilistic wind speed forecasting

Lerch, Sebastian; Thorarinsdottir, Thordis L.

doi:10.3402/tellusa.v65i0.21206

Cited by 94 publications

(117 citation statements)

References 54 publications

Supporting

Mentioning

117

Contrasting

Order By: Relevance

“…When focusing on specific sub-groups of observations, the threshold-weighted Continuous Rank Probability Score (CRPS t ) (Gneiting and Ranjan, 2011;Lerch, 2012) can be an useful criterion as it can be conditioned on different discharge signatures (similar to weather regimes see Lerch and Thorarinsdottir, 2013). It is defined by:…”

Section: Evaluation Of Benchmarksmentioning

confidence: 99%

How do I know if my forecasts are better? Using benchmarks in hydrological ensemble prediction

Pappenberger

Ramos²,

Cloke

et al. 2015

Journal of Hydrology

168

152

View full text Add to dashboard Cite

s u m m a r yThe skill of a forecast can be assessed by comparing the relative proximity of both the forecast and a benchmark to the observations. Example benchmarks include climatology or a naïve forecast. Hydrological ensemble prediction systems (HEPS) are currently transforming the hydrological forecasting environment but in this new field there is little information to guide researchers and operational forecasters on how benchmarks can be best used to evaluate their probabilistic forecasts. In this study, it is identified that the forecast skill calculated can vary depending on the benchmark selected and that the selection of a benchmark for determining forecasting system skill is sensitive to a number of hydrological and system factors. A benchmark intercomparison experiment is then undertaken using the continuous ranked probability score (CRPS), a reference forecasting system and a suite of 23 different methods to derive benchmarks. The benchmarks are assessed within the operational set-up of the European Flood Awareness System (EFAS) to determine those that are 'toughest to beat' and so give the most robust discrimination of forecast skill, particularly for the spatial average fields that EFAS relies upon.Evaluating against an observed discharge proxy the benchmark that has most utility for EFAS and avoids the most naïve skill across different hydrological situations is found to be meteorological persistency. This benchmark uses the latest meteorological observations of precipitation and temperature to drive the hydrological model. Hydrological long term average benchmarks, which are currently used in EFAS, are very easily beaten by the forecasting system and the use of these produces much naïve skill. When decomposed into seasons, the advanced meteorological benchmarks, which make use of meteorological observations from the past 20 years at the same calendar date, have the most skill discrimination. They are also good at discriminating skill in low flows and for all catchment sizes. Simpler meteorological benchmarks are particularly useful for high flows. Recommendations for EFAS are to move to routine use of meteorological persistency, an advanced meteorological benchmark and a simple meteorological benchmark in order to provide a robust evaluation of forecast skill. This work provides the first comprehensive evidence on how benchmarks can be used in evaluation of skill in probabilistic hydrological forecasts and which benchmarks are most useful for skill discrimination and avoidance of naïve skill in a large scale HEPS. It is recommended that all HEPS use the evidence and methodology provided here to evaluate which benchmarks to employ; so forecasters can have trust in their skill evaluation and will have confidence that their forecasts are indeed better.

show abstract

Section: Evaluation Of Benchmarksmentioning

confidence: 99%

How do I know if my forecasts are better? Using benchmarks in hydrological ensemble prediction

Pappenberger

Ramos²,

Cloke

et al. 2015

Journal of Hydrology

168

152

View full text Add to dashboard Cite

show abstract

“…The CRPS was devised by Epstein () and applied to meteorology by Murphy (); this score measures how well a

{normalC normalD normalF}_{i}^{f}

predicts observed behaviour by evaluating how closely it fits the cumulative distribution of the observation (denoted by a Heaviside function). Recent publications by Lerch and Thorarinsdottir () and Baran and Lerch () evaluate the ability with which more extreme events are forecast by applying a score developed by Gneiting and Ranjan () which proposes an approach that combines the CRPS with a threshold weighting function (twCRPS i ) given by:

{twCRPS}_{i} = true {truefalse\int}_{- \infty}^{\infty} {\{{normalC normalD normalF}_{i}^{normalf} (z) - H (z - y_{i})\}}^{2} w ((), z) normald z

where y i is the observed value, z is the forecast value,

H ((), z - y_{i}) = ({, \begin{matrix} center & 1 \forall z > y_{i} \\ center & 0 \forall z \leq y_{i} \end{matrix})

and w ( z ) is a (yet to be defined) non‐negative weighting function ( H ( y i – z ) replaces H ( z – y i ) for T min ). The twCRPS ranges from 0 to ∞; however, the use of w ( z ) gives the twCRPS i the ability to measure how well a forecast correctly predicts a particular type (or types) of event.…”

Section: Verification Methodologymentioning

confidence: 99%

How well do Met Office post‐processed site‐specific probabilistic forecasts predict relative‐extreme events?

Sharpe

Bysouth

Stretton

2017

Meteorological Applications

View full text Add to dashboard Cite

ABSTRACT:The Met Office routinely generates post-processed forecasts at sites throughout the United Kingdom; both deterministic and probabilistic products exist and deterministic data populate the publicly available website. In recent years, providers of weather information have focused upon the impact of events; impact is often related to the frequency of occurrence of an event at a site which is determined by its climatology. The ability with which a site-specific forecast predicts relative-extremes may be investigated by examining the skill with which these events (defined in terms of a percentile chosen from the climatology at each site) are predicted. The blended, deterministic, website forecast is less likely to forecast extreme events; therefore, the probabilistic forecast product (which does not currently appear on the Met Office website) was evaluated for its ability to predict heavy rainfall (RF 24 ), maximum summer day time temperature (T max ), minimum winter night time temperature (T min ) and strong winds (WS hrly ) over a 21 month period between December 2013 and August 2015. To this end, four methods of verification are considered: the Symmetric Extremal Dependency Index (SEDI), a threshold weighted version of the continuous ranked probability skill score (CRPSS) and a conditioned version of the CRPSS together with an analysis of the discrimination and reliability. Each method indicates forecast skill, with T max and RF 24 identified as the most and least skilful respectively and WS hrly identified as the most reliable. Site-specific values of both versions of the CRPSS appear relatively well correlated and these scores also show correlation with SEDI for WS hrly.

show abstract

“…Obviously, ω ( y ) ≡ 1 yields the traditional CRPS defined by , while one may set

ω (y) = {double-struck⊮}_{{y \geq r}}

to address wind speeds above a given threshold r . Similar to Lerch and Thorarinsdottir () and Baran and Lerch (), where the upper tail behaviors of regime‐switching EMOS models are investigated, we consider threshold values approximately corresponding to the 90th, 95th, and 99th percentiles of the wind speed observations. One can also quantify the improvement in twCRPS with respect to some reference predictive CDF F r e f with the help of the threshold‐weighted continuous ranked probability skill score (twCRPSS; e.g., Lerch and Thorarinsdottir, ) defined as follows

twCRPSS (() F, x ()) : = 1 - \frac{twCRPS (() F, x ())}{twCRPS (() F_{ref}, x ())}

This score is obviously positively oriented, and in this study, the predictive CDF corresponding to the classical TN model is used as a reference.…”

Section: Ensemble Model Output Statisticsmentioning

confidence: 99%

Mixture EMOS model for calibrating ensemble forecasts of wind speed

Baran

Lerch

2016

Environmetrics

View full text Add to dashboard Cite

Ensemble model output statistics (EMOS) is a statistical tool for post‐processing forecast ensembles of weather variables obtained from multiple runs of numerical weather prediction models in order to produce calibrated predictive probability density functions. The EMOS predictive probability density function is given by a parametric distribution with parameters depending on the ensemble forecasts. We propose an EMOS model for calibrating wind speed forecasts based on weighted mixtures of truncated normal (TN) and log‐normal (LN) distributions where model parameters and component weights are estimated by optimizing the values of proper scoring rules over a rolling training period. The new model is tested on wind speed forecasts of the 50 member European Centre for Medium‐range Weather Forecasts ensemble, the 11 member Aire Limitée Adaptation dynamique Développement International‐Hungary Ensemble Prediction System ensemble of the Hungarian Meteorological Service, and the eight‐member University of Washington mesoscale ensemble, and its predictive performance is compared with that of various benchmark EMOS models based on single parametric families and combinations thereof. The results indicate improved calibration of probabilistic and accuracy of point forecasts in comparison with the raw ensemble and climatological forecasts. The mixture EMOS model significantly outperforms the TN and LN EMOS methods; moreover, it provides better calibrated forecasts than the TN–LN combination model and offers an increased flexibility while avoiding covariate selection problems. © 2016 The Authors Environmetrics Published by JohnWiley & Sons Ltd.

show abstract

Comparison of non-homogeneous regression models for probabilistic wind speed forecasting

Cited by 94 publications

References 54 publications

How do I know if my forecasts are better? Using benchmarks in hydrological ensemble prediction

How do I know if my forecasts are better? Using benchmarks in hydrological ensemble prediction

How well do Met Office post‐processed site‐specific probabilistic forecasts predict relative‐extreme events?

Mixture EMOS model for calibrating ensemble forecasts of wind speed

Contact Info

Product

Resources

About