Observational probability method to assess ensemble precipitation forecasts

Abstract

It is common practice when assessing the skill of either deterministic or ensemble forecasts to consider the observations with no uncertainty. Observation uncertainty may be associated with different causes and the present paper discusses the uncertainty that derives from the mismatch between model-generated grid point precipitation and locally measured precipitation values. There have been many attempts to add uncertainty to the verification process; in the present paper the uncertainty is derived from the observed precipitation distribution within grid boxes of assigned resolution. The Brier skill score ( The results show that the resolution component of the BSS improves when using the O-OP method, i.e. forecast probabilities are distinguished from climatological probabilities and therefore the system has better skill. The reliability component, on the contrary, greatly degrades and this degradation is worse for lower precipitation thresholds. The results also show that the more asymmetric the precipitation distribution is within the grid box, the larger is the degradation of the reliability component. The overall BSS improves except for low thresholds. These results encourage further research into observation uncertainty and how it can be effectively accounted for in the verification of weather parameters such as precipitation.

Measuring forecast performance in the presence of observation error

“…For other authors, the verifying distribution is a distribution of observations. This might be formed from a collection of actual observations, as in Gorgas and Dorninger (2012) and Santos and Ghelli (2012). In contrast, Candille and Talagrand (2008), Pappenberger et al (2009) and Pinson and Hagedorn (2012) form the verifying distribution by randomly perturbing an observation according to a probability model of observation error.…”

“…Other authors measure the difference between f and g with a divergence, that is a function, d ( f , g ), for which d ( g , g )=0 and d ( f , g )⩾0 for all f and g . For example, Candille and Talagrand (2008) use the quadratic divergence, ( f − g ) 2 , in the case of forecasting a binary event (also Santos and Ghelli, 2012), Pappenberger et al (2009) use the Kullback–Leibler divergence (or relative entropy), , and Friederichs and Thorarinsdottir (2012) propose the integrated quadratic distance, . Thorarinsdottir et al (2013) list several other divergences, including the sub‐class of ‘score divergences’ that are formed from proper scoring rules, s , in the following way: where y ∼ g .…”

Progress and challenges in forecast verification

“…Methods are needed to account for, and ideally remove, the impact of observation errors on the verification results. Some small progress has been made in the areas of categorical (Bowler, 2006) and ensemble verification (Saetra et al ., 2004;Bowler, 2008;Candille and Talagrand, 2008;Santos and Ghelli, 2012), but this is proving a difficult problem to solve more generally. A promising approach may be to treat observations probabilistically, assuming the observation uncertainty is known (e.g., Friederichs et al ., 2009).…”

Observation‐based evaluation of ensemble reliability

“…Previous studies point out the importance of taking into account observation errors in ensemble verification (e.g. Saetra et al, 2004;Bowler, 2008;Candille and Talagrand, 2008;Santos and Ghelli, 2012). As the forecast performance improves, the impact of observation errors on the verification becomes nonnegligible, especially at short lead times.…”