Standard assessments of climate forecast skill can be misleading

Risbey, James S.; Squire, Dougal T.; Black, Amanda S.; DelSole, Timothy; Lepore, Chiara; Matear, Richard J.; Monselesan, Didier; Moore, Thomas; Richardson, Doug; Schepen, Andrew; Tippett, Michael K.; Tozer, Carly R.

doi:10.1038/s41467-021-23771-z

Cited by 44 publications

(39 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…None of the runs show rain at my location in 10 days time at 4pm, so the "chance of rain" is given as 0%. However, comparisons between models and out-of-sample outcomes show that model-based weather predictions at these lead times are not fully reliable (see, e.g., Risbey et al [2021]).…”

Section: Model Frequencies Misinterpreted As Real-world Probabilitiesmentioning

confidence: 99%

“…Where, as in seasonal climate forecasts, relevant out-of-sample testing is possible (data quality is high) but only a small amount of data is available (data quantity is low), similar trials can be undertaken using formal measures of reliability, but statistical confidence in the assessment will be lower. Additional forecast-outcome data may be generated using past data/conditions ("hindcasts") and these can provide a good quantitative measure of reliability, though with the caveat that they are not truly out-of-sample even where rigorous cross-validation approaches are employed [Risbey et al, 2021].…”

Section: Repeated Quantitative Evaluation Of Past Probabilistic Forecastsmentioning

confidence: 99%

“…For most climate forecasts, we have little out-of-sample data on the relevant timescales. While, as noted above, seasonal climate forecasts are tested against a small amount of such data, it remains unclear what skill such forecasts have [Weisheimer and Palmer, 2014;Risbey et al, 2021]. On decadal and longer timescales, out-of-sample testing is even more limited, though it is used Sherwood, 2015, Hausfather et al, 2020].…”

Section: Repeated Quantitative Evaluation Of Past Probabilistic Forecastsmentioning

confidence: 99%

“…Moreover, in climate forecasts on all timescales of interest, we are unsure about the similarity of the testing conditions and the conditions obtaining in the future [Baumberger et al, 2018;Lee and Marotzke, 2021]. This, in combination with theoretical understanding of the system's nonlinearities, gives reason to doubt the reliability of bias correction methods [Risbey et al, 2021]. Therefore, we need to fall back on theoretical evaluation of model output or on more qualitative arguments about model quality and adequacy for purpose.…”

Section: Repeated Quantitative Evaluation Of Past Probabilistic Forecastsmentioning

confidence: 99%

“…A "Model Land" PDF can, to begin with, be constructed from a model by running it multiple times with slightly different inputs and then treating the resulting frequency of outcomes as a sample from a probability distribution which can be inferred. Initial condition ensembles are used in weather and near-term climate forecasting, for example, to capture the measurement uncertainty over initial conditions and propagate this through into a PDF output [Risbey et al, 2021]. Sampling of initial conditions is in practice usually extremely limited on longer climate scales, with, for example, individual CMIP models typically contributing fewer than 10 members to CMIP ensembles [Milinski et al, 2020].…”

Section: Theory-based Evaluation Of Pdf Credibilitymentioning

confidence: 99%

See 4 more Smart Citations

On the appropriate and inappropriate uses of probability distributions in climate projections and some alternatives

et al. 2021

View full text Add to dashboard Cite

When do probability distribution functions (PDFs) about future climate misrepresent uncertainty? How can we recognise when such misrepresentation occurs and thus avoid it in reasoning about or communicating our uncertainty? And when we should not use a PDF, what should we do instead? In this paper we address these three questions. We start by providing a classification of types of uncertainty and using this classification to illustrate when PDFs misrepresent our uncertainty in a way that may adversely affect decisions. We then discuss when it is reasonable and appropriate to use a PDF to reason about or communicate uncertainty about climate. We consider two perspectives on this issue. On one, which we argue is preferable, available theory and evidence in climate science basically excludes using PDFs to represent our uncertainty. On the other, PDFs can legitimately be provided when resting on appropriate expert judgement and recognition of associated risks. Once we have specified the border between appropriate and inappropriate uses of PDFs, we explore alternatives to their use. We briefly describe two formal alternatives, namely imprecise probabilities and possibilistic distribution functions, as well as informal possibilistic alternatives. We suggest that the possibilistic alternatives are preferable.

show abstract

Section: Model Frequencies Misinterpreted As Real-world Probabilitiesmentioning

confidence: 99%

Section: Repeated Quantitative Evaluation Of Past Probabilistic Forecastsmentioning

confidence: 99%

Section: Repeated Quantitative Evaluation Of Past Probabilistic Forecastsmentioning

confidence: 99%

Section: Repeated Quantitative Evaluation Of Past Probabilistic Forecastsmentioning

confidence: 99%

Section: Theory-based Evaluation Of Pdf Credibilitymentioning

confidence: 99%

See 3 more Smart Citations

On the appropriate and inappropriate uses of probability distributions in climate projections and some alternatives

et al. 2021

View full text Add to dashboard Cite

show abstract

Predictive performance of ensemble hydroclimatic forecasts: Verification metrics, diagnostic plots and forecast attributes

Huang

Zhao

2022

WIREs Water

View full text Add to dashboard Cite

Predictive performance is one of the most important issues for practical applications of ensemble hydroclimatic forecasts. While different forecasting studies tend to use different combinations of verification metrics and diagnostic plots, this paper presents a literature survey of popular verification metrics, diagnostic plots and their corresponding forecast attributes. Twenty metrics to quantify predictive performance of ensemble, deterministic and categorical forecasts are detailed; six types of diagnostic plots to visualize the relationship between ensemble forecasts and observations are presented; and the correspondence relationships of verification metrics and diagnostic plots with ten forecast attributes are illustrated. Numerical experiments are devised for raw ensemble precipitation forecasts generated by the Climate Forecast System version 2 (CFSv2) to illustrate the verification metrics, diagnostic plots and forecast attributes. The results suggest that verification metrics that pay attention to some similar attributes can be related to one another. Meanwhile, the metrics are not necessarily in full accordance as they tend to reflect different attributes. Time series plots visualize forecasts over a period of interest and graphically illustrate forecast bias, reliability and skill. Quantile range plots provide an overall diagnosis for the association of ensemble forecasts with observations. Overall, verification metrics can be illustrated by diagnostic plots for individual grid cells and also be screened by spatial plots at regional and global scales. The survey presented in this article can serve as a roadmap for the selection of verification metrics, diagnostic plots and forecast attributes for verifying ensemble hydroclimatic forecasts. This article is categorized under: Science of Water > Hydrological Processes

show abstract

Artificial intelligence for climate prediction of extremes: State of the art, challenges, and future perspectives

Materia,

García,

van Straaten

et al. 2024

WIREs Climate Change

View full text Add to dashboard Cite

Extreme events such as heat waves and cold spells, droughts, heavy rain, and storms are particularly challenging to predict accurately due to their rarity and chaotic nature, and because of model limitations. However, recent studies have shown that there might be systemic predictability that is not being leveraged, whose exploitation could meet the need for reliable predictions of aggregated extreme weather measures on timescales from weeks to decades ahead. Recently, numerous studies have been devoted to the use of artificial intelligence (AI) to study predictability and make climate predictions. AI techniques have shown great potential to improve the prediction of extreme events and uncover their links to large‐scale and local drivers. Machine and deep learning have been explored to enhance prediction, while causal discovery and explainable AI have been tested to improve our understanding of the processes underlying predictability. Hybrid predictions combining AI, which can reveal unknown spatiotemporal connections from data, with climate models that provide the theoretical foundation and interpretability of the physical world, have shown that improving prediction skills of extremes on climate‐relevant timescales is possible. However, numerous challenges persist in various aspects, including data curation, model uncertainty, generalizability, reproducibility of methods, and workflows. This review aims at overviewing achievements and challenges in the use of AI techniques to improve the prediction of extremes at the subseasonal to decadal timescale. A few best practices are identified to increase trust in these novel techniques, and future perspectives are envisaged for further scientific development.This article is categorized under: Climate Models and Modeling > Knowledge Generation with Models The Social Status of Climate Change Knowledge > Climate Science and Decision Making

show abstract

Standard assessments of climate forecast skill can be misleading

Cited by 44 publications

References 58 publications

On the appropriate and inappropriate uses of probability distributions in climate projections and some alternatives

On the appropriate and inappropriate uses of probability distributions in climate projections and some alternatives

Predictive performance of ensemble hydroclimatic forecasts: Verification metrics, diagnostic plots and forecast attributes

Artificial intelligence for climate prediction of extremes: State of the art, challenges, and future perspectives

Contact Info

Product

Resources

About