Optimal Weight Choice for Frequentist Model Average Estimators

Abstract

There has been increasing interest recently in model averaging within the frequentist paradigm. The main benefit of model averaging over model selection is that it incorporates rather than ignores the uncertainty inherent in the model selection process. One of the most important, yet challenging, aspects of model averaging is how to optimally combine estimates from different models. In this work, we suggest a procedure of weight choice for frequentist model average estimators that exhibits optimality properties with respect to the estimator's mean squared error (MSE). As a basis for demonstrating our idea, we consider averaging over a sequence of linear regression models. Building on this base, we develop a model weighting mechanism that involves minimizing the trace of an unbiased estimator of the model average estimator's MSE. We further obtain results that reflect the finite sample as well as asymptotic optimality of the proposed mechanism. A Monte Carlo study based on simulated and real data evaluates and compares the finite sample properties of this mechanism with those of existing methods. The extension of the proposed weight selection scheme to general likelihood models is also considered. This article has supplementary material online.

Minimum Mean Squared Error Model Averaging in Likelihood Models

“…The narrow model corresponds to S = ∅. Since our method of finding weights is based on minimizing a mean squared error expression, see also Liang et al (2011), this setting is justified since it balances the squared bias and the variance of the estimators in order for the mean squared error to be computable. Indeed, when not working under local misspecification, for a fixed true model not contained in the set of studied models asymptotically the bias would dominate, pointing toward always working with the most complicated model (Claeskens and Hjort (2008)).…”

“…We investigate the finite sample performance of the minimum MSE estimator (mMSE) and compare the results with other methods of averaging, in particular by the plug-in estimator (Liu (2015)), the so-called optimal estimator (OPT, Liang et al (2011)), Mallows model averaging (MMA, Hansen (2007)) and jackknife model averaging (JMA, Hansen and Racine (2012)). All of these methods are defined for averaging over all possible submodels.…”

“…While selecting a model based on the estimator's estimated MSE value is the idea underlying the focused information criterion ), we here consider the choice of the weights via the mean squared error, similar as Liu (2015) and Liang et al (2011), though for general likelihood estimation and a general choice of weights summing to one.…”

“…This method can be considered as an extension of Liu (2015) to likelihood models. We also extend the approach of Liang et al (2011) as we do not restrict the empirical weights to have a certain parametric form nor to lie in the unit simplex, although we impose that the sum of weights is equal to one as is necessary for consistency of the model averaging estimator ). In absence of imposing inequality restrictions, unlike other weight selection methods, no quadratic programming or nonlinear optimization is required.…”

“…For a further literature overview, see Cheng and Hansen (2015). Liang et al (2011) proposed to select the weights such that the estimated MSE of the weighted estimatorμ w is minimal. In that paper, their 'optimal' set of weights for frequentist model-averaged estimators is, however, restricted to a specific ad hoc parametric form.…”