Spatial-temporal fractions verification for high-resolution ensemble forecasts

Abstract

A B S T R A C T Experiments with two ensemble systems of resolutions 10 km (MF10km) and 2 km (MF2km) were designed to examine the value of cloud-resolving ensemble forecast in predicting precipitation on small spatio-temporal scales. Since the verification was performed on short-term precipitation at high resolution, uncertainties from small-scale processes caused the traditional verification methods to be inconsistent with the subjective evaluation. An extended verification method based on the Fractions Skill Score (FSS) was introduced to account for these uncertainties. The main idea is to extend the concept of spatial neighbourhood in FSS to the time and ensemble dimension. The extension was carried out by recognising that even if ensemble forecast is used, small-scale variability still exists in forecasts and influences verification results. In addition to FSS, the neighbourhood concept was also incorporated into reliability diagrams and relative operating characteristics to verify the reliability and resolution of two systems.The extension of FSS in time dimension demonstrates the important role of temporal scales in short-term precipitation verification at small spatial scales. The extension of FSS in ensemble space is called the ensemble FSS, which is a good representative of FSS for ensemble forecast in comparison with the FSS of ensemble mean. The verification results show that MF2km outperforms MF10km in heavy rain forecasts. In contrast, MF10km was slightly better than MF2km in predicting light rains, suggesting that the horizontal resolution of 2 km is not necessarily enough to completely resolve convective cells.

On the forecast skill of a convection-permitting ensemble

“…These results are comparable to the main outcomes of Le Duc et al (2013) and Schwartz et al (2009). Le Duc et al (2013) also found that the coarser 10 km ensemble showed slightly better results for light rains than the finer 2 km one.…”

“…Le Duc et al (2013) also found that the coarser 10 km ensemble showed slightly better results for light rains than the finer 2 km one. Both models had lower skill in predicting heavy rain; however, for the higher precipitation thresholds, the 2 km ensemble performed better than the 10 km one.…”

“…Hence, ensemble-related scores are combined with spatial verification methods. Unintentionally, the strategy of this paper shows parallels to the verification study conducted by Le Duc et al (2013), especially concerning the two ensembles (10 and 2 km resolution) coupled by direct downscaling. Further similarities are the complex terrain in which the study is conducted (Japan) and the use of traditional and advanced verification metrics.…”

“…Their case studies revealed that the convection-permitting ensemble generally provided more accurate precipitation forecasts than the coarser-resolution regional EPS. Le Duc et al (2013) examined the ability to predict precipitation of two 11-member ensembles with 10 and 2 km horizontal resolution, with the fine model using direct downscaling of the coarser one. They could show that the 10 km ensemble was more reliable in predicting light rain, whereas the 2 km ensemble outperformed the coarser one in cases of heavier rain.…”

The Setup of the MesoVICT Project

“…Since the ICP special collection, a large number of papers have been published making use of the newer spatial techniques. For example, neighborhood methods [particularly the fractions skill score (FSS)] have been employed by Weusthoff et al (2010), Schaffer et al (2011), Sobash et al (2011, Duc et al (2013), Mittermaier et al (2013), and Skok and Roberts (2016). Scale separation methods have been used by De Sales andLiu et al (2011), and field deformation approaches by Nan et al (2010).…”