Model selection by resampling penalization

Abstract

In this paper, a new family of resampling-based penalization procedures for model selection is defined in a general framework. It generalizes several methods, including Efron's bootstrap penalization and the leave-one-out penalization recently proposed by Arlot (2008), to any exchangeable weighted bootstrap resampling scheme. In the heteroscedastic regression framework, assuming the models to have a particular structure, these resampling penalties are proved to satisfy a non-asymptotic oracle inequality with leading constant close to 1. In particular, they are asympotically optimal. Resampling penalties are used for defining an estimator adapting simultaneously to the smoothness of the regression function and to the heteroscedasticity of the noise. This is remarkable because resampling penalties are general-purpose devices, which have not been built specifically to handle heteroscedastic data. Hence, resampling penalties naturally adapt to heteroscedasticity. A simulation study shows that resampling penalties improve on V-fold cross-validation in terms of final prediction error, in particular when the signal-to-noise ratio is not large.Comment: extended version of hal-00125455, with a technical appendi

Segmentation of the mean of heteroscedastic data via cross-validation

“…In particular, precise concentration inequalities for R Lpo p (m) would be needed. According to the assumptions under which similar results have been obtained for polynomial collections of models (Arlot 2008(Arlot , 2009Celisse 2008b), we can conjecture that (at least) the following assumptions would be required to derive an upper bound on s − s m Lpo p (D) 2 n : • moment inequalities for the errors ε i : ∀q ≥ 2, ε q ≤ Cq β for some C, β > 0 (for instance, ε sub-Gaussian), • a uniform upper bound on σ (see (BN) in Proposition 1), and probably a uniform lower bound or mild smoothness assumptions on σ (·), • a relationship between the minimum number of points in the intervals of a segmentation and the smoothness of the regression function s. Note that assumption (BV) in Proposition 1 entails that for every λ ∈ m and t i ∈ λ,…”

“…In particular, Loo is asymptotically equivalent to AIC or C p in several frameworks where they are asymptotically optimal, and other CV methods have similar performances, provided the size of the training sample is close enough to the sample size (see for instance Li 1987;Shao 1997;Dudoit and van der Laan 2005). In addition, CV methods are robust to heteroscedasticity of data (Arlot 2008), as well as several other resampling methods (Arlot 2009). Therefore, CV is a natural alternative to penalization procedures assuming homoscedasticity.…”

“…Therefore, other procedures must be used. For instance, resampling penalization is optimal with heteroscedastic data and polynomial collections (Arlot 2009). Another approach has been explored by Comte and Rozenholc (2002) and by Gendre (2008), which consists in simultaneously estimating the mean and the variance, using polynomial collections of models.…”

Adaptive partitioning schemes for bipartite ranking

“…However, complexity-based penalties cannot be used straightforwardly since further studies are needed to calibrate the numerical constants involved. We refer to a recent paper by Arlot (2009) where resampling strategies are depicted in the classification setup in order to address this issue. The simulation study we performed reveals the potential but also the limitations of these scoring algorithms.…”

Data Verifications for Online Social Networks

“…We ran K-means clustering [Bis95] on the free-form chunk dataset, to cluster the chunks according to their motion (see Cluster attack). We applied the v-fold cross-validation algorithm [Arl07] to determine the optimal number of clusters in our dataset. The outcome was K = 6.…”