A Generic Method for Density Forecasts Recalibration

Collet, Jérôme; Richard, Michael D.

doi:10.1007/978-3-319-99052-1_8

Cited by 3 publications

(10 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Both performance criteria should be used. Focussing only on the CRPS may lead to choosing a forecasting system that is not optimal in terms of reliability as shown here and earlier studies (Collet & Richard, 2017; Wilks, 2018). But relying only on the JP flatness tests may also be misleading.…”

Section: Discussion About Probabilistic Forecast Selectionmentioning

confidence: 76%

“…The second approach to model selection among probabilistic forecasting systems is based on a scoring rule, such as the continuous ranked probabilistic score (CRPS, Matheson and Winkler, 1976): the selected model is the one that has the best value of the scoring rule (highest or lowest value depending on the scoring rule). The two approaches to model selection do not yield equivalent forecasts, as previously mentioned in different studies (Collet & Richard, 2017; Wilks, 2018). For instance, minimizing the CRPS may lead to forecasts that are not reliable.…”

Section: Introductionmentioning

confidence: 82%

“…Collet and Richard (2017) proposed a post‐processing method that enforces the reliability of the forecast, by using the probability integral transform (PIT). To have a more general forecast selection method, it is proposed here to check that a forecasting system is reliable by testing the hypothesis that its rank histogram is flat.…”

Section: Theoretical Framework and Performance Assessment Toolsmentioning

confidence: 99%

“…The second goal of this work is to compare two model selection approaches when dealing with probabilistic forecasting systems (Collet & Richard, 2017). The first approach is based on ‘reliability’ (or ‘calibration’) and ‘sharpness’ (Gneiting et al., 2007; Jolliffe & Stephenson, 2011).…”

Section: Introductionmentioning

confidence: 99%

“…To solve this problem, Wilks (2018) proposes to minimize the CRPS penalized with a term quantifying the unreliability of the forecast. Collet and Richard (2017) introduces a post‐processing method that, under quite strong assumptions, yields reliable forecasts without degrading too much the CRPS compared to the CRPS‐minimizing approach. We do not use any of these solutions here but compare the two model selection approaches and their properties on a case study.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Sequential Aggregation of Probabilistic Forecasts—Application to Wind Speed Ensemble Forecasts

Zamo

Bel

Mestre

2021

Journal of the Royal Statistical Society Series C: Applied Statistics

View full text Add to dashboard Cite

In numerical weather prediction (NWP), the uncertainty about the future state of the atmosphere is described by a set of forecasts (called an ensemble). All ensembles have deficiencies that can be corrected via statistical post‐processing methods. Several ensembles, based on different NWP models, exist and may be corrected using different statistical methods. These raw or post‐processed ensembles can thus be combined. The theory of prediction with expert advice allows us to build combination algorithms with theoretical guarantees on the forecast performance. We adapt this theory to the case of probabilistic forecasts issued as stepwise cumulative distribution functions, computed from raw and post‐processed ensembles. The theory is applied to combine wind speed ensemble forecasts. The second goal of this study is to explore the use of two forecast performance criteria: the continuous ranked probability score (CRPS) and the Jolliffe–Primo test. The usual way to build skilful probabilistic forecasts is to minimize the CRPS. Minimizing the CRPS may not produce reliable forecasts according to the Jolliffe–Primo test. The Jolliffe–Primo test generally selects reliable forecasts, but could lead to issuing suboptimal forecasts in terms of CRPS. We propose to use both criteria to achieve reliable and skilful probabilistic forecasts.

show abstract

Section: Discussion About Probabilistic Forecast Selectionmentioning

confidence: 76%

Section: Introductionmentioning

confidence: 82%

Section: Theoretical Framework and Performance Assessment Toolsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Sequential Aggregation of Probabilistic Forecasts—Application to Wind Speed Ensemble Forecasts

Zamo

Bel

Mestre

2021

Journal of the Royal Statistical Society Series C: Applied Statistics

View full text Add to dashboard Cite

show abstract

Sequential Aggregation of Probabilistic Forecasts -- Applicaton to Wind Speed Ensemble Forecasts

Zamo¹,

Bel²,

Mestre³

2020

Preprint

View full text Add to dashboard Cite

In the field of numerical weather prediction (NWP), the probabilistic distribution of the future state of the atmosphere is sampled with Monte-Carlo-like simulations, called ensembles. These ensembles have deficiencies (such as conditional biases) that can be corrected thanks to statistical post-processing methods. Several ensembles exist and may be corrected with different statistiscal methods. A further step is to combine these raw or post-processed ensembles. The theory of prediction with expert advice allows us to build combination algorithms with theoretical guarantees on the forecast performance. This article adapts this theory to the case of probabilistic forecasts issued as step-wise cumulative distribution functions (CDF). The theory is applied to wind speed forecasting, by combining several raw or post-processed ensembles, considered as CDFs. The second goal of this study is to explore the use of two forecast performance criteria: the Continous ranked probability score (CRPS) and the Jolliffe-Primo test. Comparing the results obtained with both criteria leads to reconsidering the usual way to build skillful probabilistic forecasts, based on the minimization of the CRPS. Minimizing the CRPS does not necessarily produce reliable forecasts according to the Jolliffe-Primo test. The Jolliffe-Primo test generally selects reliable forecasts, but could lead to issuing suboptimal forecasts in terms of CRPS. It is proposed to use both criterion to achieve reliable and skillful probabilistic forecasts.

show abstract

Sequential Aggregation of Probabilistic Forecasts - Applicaton to Wind Speed Ensemble Forecasts

Zamo¹,

Bel²,

Mestre³

2021

Preprint

View full text Add to dashboard Cite

<p>Sequential aggregation is a theoretically-grounded means to combine several forecasts of a quantity to achieve better forecast performance as evaluated by a loss function. This theory has been mainly applied to point forecasts with a scalar forecast quantity, but rarely to forecasts expressed in a probabilistic form. In this work, we show how this theory can be readily adapted to forecasts expressed as step-wise cumulative distribution function (CDF), with the continuous ranked probabilistic score (CRPS) as performance measure.</p><p>Ensemble weather forecasts estimate the outcome of future observed quantities in a way that can be interpreted as step-wise CDF. Since those forecast CDFs are biased, statistical postprocessing methods are used to improve their statistical coherency with the observed quantity. Since many ensembles and many postprocessing methods exist, one can combine raw and post-processed ensembles in order to get even better forecast performance. To illustrate this point and the advantages of blending CDFs, sequential aggregation is applied to wind-speed ensemble weather forecasts with the CRPS as a performance measure alongside the Jolliffe-Primo test to assess the reliability of the various (raw, post-processed or aggregated) forecasts.</p>

show abstract

A Generic Method for Density Forecasts Recalibration

Cited by 3 publications

References 11 publications

Sequential Aggregation of Probabilistic Forecasts—Application to Wind Speed Ensemble Forecasts

Sequential Aggregation of Probabilistic Forecasts—Application to Wind Speed Ensemble Forecasts

Sequential Aggregation of Probabilistic Forecasts -- Applicaton to Wind Speed Ensemble Forecasts

Sequential Aggregation of Probabilistic Forecasts - Applicaton to Wind Speed Ensemble Forecasts

Contact Info

Product

Resources

About