A methodology for sensitivity analysis of spatial features in forecasts: the stochastic kinetic energy backscatter scheme

Abstract

Stochastic kinetic energy backscatter schemes (SKEBSs) are introduced in numerical weather forecast models to represent uncertainties related to unresolved subgrid‐scale processes. These schemes are formulated using a set of parameters that must be determined using physical knowledge and/or to obtain a desired outcome. Here, a methodology is developed for assessing the effect of four factors on spatial features of forecasts simulated by the SKEBS‐enabled Weather Research and Forecasting model. The four factors include two physically motivated SKEBS parameters (the determining amplitude of perturbations applied to stream function and potential temperature tendencies), a purely stochastic element (a seed used in generating random perturbations) and a factor reflecting daily variability. A simple threshold‐based approach for identifying coherent objects within forecast fields is employed, and the effect of the four factors on object features (e.g. number, size and intensity) is assessed. Four object types are examined: upper‐air jet streaks, low‐level jets, precipitation areas and frontal boundaries. The proposed method consists of a set of standard techniques in experimental design, based on the analysis of variance, tailored to sensitivity analysis. More specifically, a Latin square design is employed to reduce the number of model simulations necessary for performing the sensitivity analysis. Fixed effects and random effects models are employed to assess the main effects and the percentage of the total variability explained by the four factors. It is found that the two SKEBS parameters do not have an appreciable and/or statistically significant effect on any of the examined object features.

A Sensitivity Analysis of Two Mesoscale Models: COAMPS and WRF

“…A total of 50 different values for these 9 parameters are selected according to LHS; the range over which they are varied, and their default value is also shown in Table 1. LHS has been used in sampling the model parameter space (Marzban et al 2014), selecting the members in ensembles for ensemble forecasting (Hacker et al 2011), for emulation (Santner et al 2003), and for performing variance-based SA (Saltelli et al 2010(Saltelli et al , 2008. The popularity of this sampling scheme derives from the fact that it leads to estimates that are often more precise, never less, than simple random sampling (Cioppa and Lucas 2007;Marzban 2013).…”

“…Sensitivity analysis (SA) refers to a wide range of techniques for assessing the impact of something on something else (e.g., the impact of a single observation on the forecasts in the process of data assimilation, or the impact of model parameters on the forecasts). In meteorology various SA methods have been developed (Aires et al 2014;Fasso 2006;Oakley and O'Hagan 2004;Saltelli et al 2010;Sobol' 1993;Zhao and Tiede 2011;Lucas et al 2013;Errico 1997;Ancell and Hakim 2007;Safta et al 2015;Hacker et al 2011;Laine et al 2012;Ollinaho et al 2014;Roebber 1989;Roebber and Bosart 1998;Robock et al 2003;Järvinen et al 2012;Marzban 2013;Marzban et al 2014Marzban et al , 2018aMarzban et al ,b, 2019Marzban et al , 2020, and a partial taxonomy of the these methods is provided in Marzban et al (2019).…”

“…For discrete model parameters two of the most commonly used sampling designs are Graeco-Latin square designs and fractional factorial designs (Montgomery 2009). The case of continuous model parameters is covered in Marzban et al (2014Marzban et al ( , 2018a, where the forecast model is COAMPS (Hodur 1997). The case of discrete model parameters is described in Marzban et al (2019Marzban et al ( , 2020, where the model parameters are those of the stochastic kinetic energy backscatter scheme (Leith 1990;Mason and Thomson 1992;Berner et al 2011) in the Weather Research and Forecasting (WRF) Model.…”

“…The forecast fields examined are 24-h forecasts of water vapor, convective and gridscale precipitation, and air temperature and wind speed at three different vertical levels. Spatial structure is often quantified by a variogram; a graph which, among other things, also reveals the extent of spatial correlations (Marzban and Sandgathe 2009). However, by virtue of being diagrams, variograms are not suitable for displaying local spatial structure across a larger domain.…”

“…Here, suffice it to say that 50 different values of 9 model parameters are selected, multivariate multiple regression is employed to map the model parameters to forecast fields on 30 days, and regression coefficients and p values are used as measures of sensitivity. 1 Further details of the SA methodology can be found in the prequels to this article (e.g., Marzban et al 2014Marzban et al , 2018aMarzban et al ,b, 2019; the main focus of the present article is to apply that methodology to the problem of comparing COAMPS and WRF in terms of their sensitivity with respect to a common set of model parameters.…”