Information Criteria for Deciding between Normal Regression Models

Abstract

Regression models fitted to data can be assessed on their goodness of fit, though models with many parameters should be disfavored to prevent over-fitting. Statisticians' tools for this are little known to physical scientists. These include the Akaike Information Criterion (AIC), a penalized goodness-of-fit statistic, and the AICc, a variant including a small-sample correction. They entered the physical sciences through being used by astrophysicists to compare cosmological models; e.g., predictions of the distance-redshift relation. The AICc is shown to have been mis-applied, being applicable only if error variances are unknown. If error bars accompany the data, the AIC should be used instead. Erroneous applications of the AICc are listed in an appendix. It is also shown how the variability of the AIC difference between models with a known error variance can be estimated. This yields a significance test that can potentially replace the use of 'Akaike weights' for deciding between such models. Additionally, the effects of model mis-specification are examined. For regression models fitted to data sets without (rather than with) error bars, they are major: the AICc may be shifted by an unknown amount. The extent of this in the fitting of physical models remains to be studied.

Spica and the annual cycle of PKS B1322–110 scintillations

“…We experimented with the number of terms in (4) by running the optimisation code NLopt (Johnson 2010 1 , Powell 2009 2 ) for a fixed number of objective function evaluations and various N and compared them using the adjusted Akaike information criterion (e.g. Maier 2013), which produced more consistent results compared to similarly used Bayesian information criteria. In most cases the optimum value turned out to be N = 1 and it was never above four; moreover, the results for r i did not seem to be much affected if just a single term was used.…”