Sensitivity of Calibrated Week-2 Probabilistic Forecast Skill to Reforecast Sampling of the NCEP Global Ensemble Forecast System

Abstract

The final published version of this manuscript will replace the preliminary version at the above DOI once it is available.If you would like to cite this EOR in a separate work, please use the following full citation:Ou, M., M. Charles, and D. Collins, 2016: Sensitivity of calibrated week-2 probabilistic forecast skill to reforecast sampling of the NCEP Global Ensemble Forecast System. Wea. Forecasting. Abstract 21 22CPC requires the reforecast-calibrated GEFS to support the production of their 23 official 6-10 and 8-14 day temperature and precipitation forecasts. While a large sample 24 size of forecast-observation pairs is desirable to generate the necessary model 25 climatology and variances, and covariances to observations, sampling by reforecasts 26 could be done to use available computing resources most efficiently. A series of 27 experiments was done to assess the impact on calibrated forecast skill of using a 28 smaller sample size than the current available reforecast dataset. This study focuses on 29 the skill of week-2 probabilistic forecasts of the 7-day mean 2-meter temperature and 30 accumulated precipitation. The tercile forecasts are expressed as being below-, near-, 31 and above-normal temperature/median precipitation over the CONUS. Calibration 32 statistics were calculated using an ensemble regression technique from 25 years of 33 daily, 11-member GEFS reforecasts for 1986-2010, which were then used to post-34 process the GEFS model forecasts for 2011-2013. In assessing the skill of calibrated 35 model output using a reforecast dataset with fewer years and ensemble members, and 36 an ensemble run less frequently than daily, it was determined that reductions in the 37 number of ensemble members to 6 or fewer and reductions in the frequency of 38 reforecast runs from daily to once a week were achievable with minimal loss of skill. 39However, reducing the number of years of reforecasts to less than 25 resulted in a 40 greater skill degradation. The loss of skill was statistically significant using only 18 years 41 3 of reforecasts from 1993-2010 to generate model statistics. 42

Systematic Error Analysis and Calibration of 2-m Temperature for the NCEP GEFS Reforecast of the Subseasonal Experiment (SubX) Project

“…The calibration of the ensemble forecast system is evaluated via the RMSE (Zhu and Toth 2008) and rank probability skill score (RPSS; Wilks 2011). The RPSS is frequently used for evaluating the performance of probabilistic forecasts (Ou et al 2016;Melhauser et al 2016;Zhu et al 2017), which measures the improvement of a multicategory forecast relative to a reference analysis. The higher the RPSS, the more skillfully the probabilistic system performs.…”

“…The sensitivity of forecast skill to the number of training years has been studied by Hamill et al (2004), Guan et al (2015), and Ou et al (2016). Using the firstgeneration GEFS reforecast dataset, Hamill et al (2004) demonstrated that there was a significant increase in skill from 2 to 5 years of training data for week 2 surface temperature, but only showed small incremental increases by 10-12 years.…”

“…Guan et al (2015) with more skillful GEFSv10 reforecast data (Hamill et al 2013) reveal that the improvement from using a 5-yr training period is almost equivalent to the improvement seen when using a 10-or 25-yr training period for lead times up to 16 days. Using the same dataset, Ou et al (2016) showed an 18-yr training period is desirable for high quality week 2 calibration over the CONUS.…”

“…To test the sensitivity of the weeks 3 and 4 forecast skill to the number of training years, we calibrate the 2016 forecast using the 5-yr (2011-15), 10-yr (2006-15), and 17-yr (1999-2015) training datasets. These specific training years were chosen because the forecast skill in predicting 2-m temperature displays a steady increase from the 5-to 10-yr training periods and then nears saturation (Hamill et al 2004;Ou et al 2016;Guan et al 2015). Figure 14 shows that increasing the number of training years from 5 to 10 years leads to a gain in skill of 0.016 (or ;5%), whereas further increasing to 17 years does not yield additional improvement.…”

“…These techniques include Kalman filtering (Cheng and Steenburgh 2007), decaying averaging methods (Cui et al 2012), logistic regression (Wilks and Hamill 2007), nonhomogeneous Gaussian regression (Gneiting et al 2005), Gaussian ensemble dressing (Roulston and Smith 2003;Wang and Bishop 2005;Bishop and Shanley 2008), Bayesian model averaging (Raftery et al 2005;Wilson et al 2007), artificial neural networks (Yuan et al 2007), and analog techniques (Hamill et al 2013). Previous studies (Hamill et al 2004(Hamill et al , 2008Cui et al 2012;Guan et al 2015;Guan and Zhu 2017;Ou et al 2016) have revealed the importance of a hindcast (i.e., reforecast) for extreme weather forecasts or bias correction during week 1 or week 2. Thus, the hybrid decaying and reforecast bias-correction method (Guan et al 2015) is being operationally applied to the North American Ensemble Forecast System (NAEFS; Candille 2009) in order to improve 1-16-day forecasts.…”