Evaluating Automatic Model Selection

Abstract

We outline a range of criteria for evaluating model selection approaches that have been used in the literature. Focusing on three key criteria, we evaluate automatically selecting the relevant variables in an econometric model from a large candidate set. General-tospecific selection is outlined for a regression model in orthogonal variables, where only one decision is required to select, irrespective of the number of regressors. Comparisons with an automated model selection algorithm, Autometrics (Doornik, 2009), show similar properties, but not restricted to orthogonal cases. Monte Carlo experiments examine the roles of post-selection bias corrections and diagnostic testing, and evaluate selection in dynamic models by costs of search versus costs of inference.JEL classifications: C51, C22.

Modelling Financial Markets Comovements during Crises: A Dynamic Multi-Factor Approach

“…which we rewrite for convenience, as: cov n;m;t =ĉ ov F n;m;t +ĉ ov n;m;t (14) Correspondingly, de…ne the quantitiesĉorr F n;m;t andĉorr n;m;t dividing by the appropriate variances. We provide the estimates ofĉorr n;m;t via the DCC framework.…”

Model Discovery and Trygve Haavelmo’s Legacy

“…There are many ways to judge the performance of such algorithms, but some basic requirements are that the null retention rate of irrelevant variables (gauge) is close to the nominal size, α, set for selection tests; the retention rate of relevant variables (potency) is not far below the corresponding power of the equivalent test in the relevant LDGP; and that the LDGP is located almost as often starting from the postulated initial general model as when starting from the LDGP itself: see Castle, Doornik, and Hendry (2011). Gauge and potency differ from size and power respectively, not least because algorithms can retain variables whose estimated coefficients are insignificant on the selection statistic.…”

“…However, when (11) does nest the DGP, selection can improve the final model relative to (9), as in Castle, Doornik, and Hendry (2011). While retaining x t when selecting from (15) will then deliver an incorrect estimate of β 0 , relevant u i,t with large non-centralities will usually be retained, this time correctly, rejecting the validity of the theory model and providing a better approximation to the LDGP.…”

“…The approach can allow for many potentially-relevant variables, perhaps at long lags, with possibly non-linear reactions, facing multiple location shifts and outliers: see Castle, Doornik, and Hendry (2011) (who provide bibliographic perspective), Nielsen (2009), andHendry (2012). The theory model is embedded in the general specification, but is not selected over, whereas other aspects are selected at tight significance levels, so are only retained when highly significant.…”

An Experimental Study of a Common Property Renewable Resource Game in Continuous Time

“…Since there is apparent smoothness in the extraction behaviour, a multipath search for the optimal specification of a dynamic model is required (Castle, Doornik andHendry, 2011, Mizon,1995). Hendry and Krolzig have developed the PcGets software for just such an automated model selection.…”