Catching up Faster by Switching Sooner: A Predictive Approach to Adaptive Estimation with an Application to the AIC–BIC Dilemma

Abstract

Summary. Prediction and estimation based on Bayesian model selection and model averaging, and derived methods such as the Bayesian information criterion BIC, do not always converge at the fastest possible rate. We identify the catch-up phenomenon as a novel explanation for the slow convergence of Bayesian methods, which inspires a modification of the Bayesian predictive distribution, called the switch distribution. When used as an adaptive estimator, the switch distribution does achieve optimal cumulative risk convergence rates in non-parametric density estimation and Gaussian regression problems. We show that the minimax cumulative risk is obtained under very weak conditions and without knowledge of the underlying degree of smoothness. Unlike other adaptive model selection procedures such as the Akaike information criterion AIC and leave-one-out cross-validation, BIC and Bayes factor model selection are typically statistically consistent. We show that this property is retained by the switch distribution, which thus solves the AIC-BIC dilemma for cumulative risk. The switch distribution has an efficient implementation. We compare its performance with AIC, BIC and Bayesian model selection and averaging on a regression problem with simulated data.

A Bayesian Information Criterion for Singular Models

“…Information criteria are valuable tools for model selection (Burnham and Anderson, ; Claeskens and Hjort, ; Konishi and Kitagawa, ). At a high level, they fall into two categories (Yang, ; van Erven et al ., ; Wit et al ., ). On one side, there are criteria that target good predictive behaviour of the selected model.…”

Luckiness and Regret in Minimum Description Length Inference

“…However as conjectured in [19], it can probably be extended to other universal codes on the inner level, such as NML (for some special cases, it was proven in [41] that this is indeed the case). Note that (as we keep emphasising) we do not make any assumptions about the true distribution P * .…”

“…For example, in many situations, one can achieve both consistency and optimal convergence rates; see Section 4.2.2. For more details, please refer to [41].…”

A tutorial on adaptive design optimization

“…That is, it has been shown that the model with the highest posterior probability is also the one with the smallest accumulative prediction error even when none of the models under consideration is the data-generating model (Kass and Raftery, 1995; Wagenmakers et al, 2006; van Erven et al, 2012). …”