Minimal Penalties for Gaussian Model Selection

Abstract

This paper is mainly devoted to a precise analysis of what kind of penalties should be used in order to perform model selection via the minimization of a penalized least-squares type criterion within some general Gaussian framework including the classical ones. As compared to our previous paper on this topic (Birgé and Massart in J. Eur. Math. Soc. 3, 203-268 (2001)), more elaborate forms of the penalties are given which are shown to be, in some sense, optimal. We indeed provide more precise upper bounds for the risk of the penalized estimators and lower bounds for the penalty terms, showing that the use of smaller penalties may lead to disastrous results. These lower bounds may also be used to design a practical strategy that allows to estimate the penalty from the data when the amount of noise is unknown. We provide an illustration of the method for the problem of estimating a piecewise constant signal in Gaussian noise when neither the number, nor the location of the change points are known.

Information criteria for variable selection under sparsity

“…Simulations discussed in the Supplementary Material show that the mirror penalty detects the degree of sparsity automatically. It can be shown that for n 1 larger than that degree, the mirror penalty ν(n 1 ) increases faster than the lower bound of Birgé and Massart (2007). Unlike that lower bound, however, the mirror paradigm is not limited to normal errors or to Mallows' C p criterion.…”

“…The mirror corrected penalty can be compared to the minimum penalty for consistent estimators (Birgé and Massart, 2007). Being a lower bound, that penalty is not data-specific, unlike that proposed in this paper.…”

“…Supplementary Material includes a proof of Proposition 1, the study of the mirror effect for Akaike's information criterion, a few interpretations on the proofs in Appendix A, a discussion on mirror penalties versus penalties proposed in Birgé and Massart (2007), a discussion on Assumption 2, and a note on accompanying software.…”

“…An important benchmark in this respect is the minimum penalty resulting from the analysis by Birgé and Massart (2007).…”

“…This paper discusses the application for both Mallows' C p and Akaike's Information Criterion (Akaike, 1973). In the case of normal errors and Mallows' C p , the resulting penalty term can be compared to a lower bound that avoids inconsistent estimators (Birgé and Massart, 2007). The mirror correction, being data-dependent, automatically finds the degree of sparsity in the given data.…”