Model selection for regression on a random design

Baraud, Yannick

doi:10.1051/ps:2002007

Cited by 86 publications

(129 citation statements)

References 17 publications

Supporting

Mentioning

124

Contrasting

Order By: Relevance

“…Some illustrations of these difficulties can be found in most papers and books on the subject, among which (small sample) van de Geer (1993 and, Birgé and Massart (1997), Barron, Birgé and Massart (1999), Castellan (1999 and or Wegkamp (2003). Further limitations of the classical methods for model selection are connected with the choice of the models which have to share some special properties: they should, for instance, be finite dimensional linear spaces generated by special bases, as in Baraud (2002) or be uniformly bounded as in Yang (2000).…”

Section: Some Motivationsmentioning

confidence: 99%

Model selection via testing: an alternative to (penalized) maximum likelihood estimators

Birgé¹

2006

Annales de l'Institut Henri Poincare (B) Probability and Statis

189

View full text Add to dashboard Cite

This paper is devoted to the description and study of a family of estimators, that we shall call T -estimators (T for tests), for minimax estimation and model selection. Their construction is based on former ideas about deriving estimators from some families of tests due to Le Cam (1973 and1975) and Birgé (1983, 1984a and and about complexity based model selection from Barron and Cover (1991).It is well-known that maximum likelihood estimators or, more generally, minimum contrast estimators do suffer from various weaknesses, and their penalized versions as well. In particular they are not robust and they require restrictive assumptions on both the models and the underlying parameter for the estimators to work correctly. Our method, which derives an estimator from many simultaneous tests between some probability balls in a suitable metric space, tends to solve many of these difficulties. Its robustness properties allow to deal with minimax estimation and model selection in a unified way, since bounding the minimax risk amounts to perform the method with a single, well-chosen, model. This results in simple bounds for the minimax risk solely based on some metric properties of the parameter space. Moreover the method applies to various statistical frameworks (we shall concentrate here on the i.i.d. and the Gaussian sequence settings only) and can handle essentially all types of models, linear or not, parametric and non-parametric, simultaneously. From these viewpoints, it is much more flexible than traditional methods. In particular, we shall be able to derive some adaptation results for density estimation over Besov balls that do not seem to be accessible to classical methods. The counterpart for these nice properties is the very high computational complexity of our construction which makes our estimators look more like theoretical than practical tools. 0 AMS 1991 subject classifications. Primary 62F35; secondary 62G05.

show abstract

Section: Some Motivationsmentioning

confidence: 99%

Model selection via testing: an alternative to (penalized) maximum likelihood estimators

Birgé¹

2006

Annales de l'Institut Henri Poincare (B) Probability and Statis

189

View full text Add to dashboard Cite

show abstract

“…Focusing on the simplest situation of Gaussian settings allows to describe the main specificities of our method with less technical efforts, to better emphasize the ideas underlying our approach and to get much more precise results with shorter proofs. Generalizations have been developed for general regression (possibly non-Gaussian) settings by Baraud (1997 and and Baraud et al (1997 and and for exponential models in density estimation by Castellan (1999). A penalization method based on model complexity and which is close to ours can be found in Yang (1999).…”

Section: Introducing Model Selection From a Nonasymptotic Point Of Viewmentioning

confidence: 99%

Gaussian model selection

Birgé¹,

Massart²

2001

J. Eur. Math. Soc.

367

442

View full text Add to dashboard Cite

Abstract. Our purpose in this paper is to provide a general approach to model selection via penalization for Gaussian regression and to develop our point of view about this subject. The advantage and importance of model selection come from the fact that it provides a suitable approach to many different types of problems, starting from model selection per se (among a family of parametric models, which one is more suitable for the data at hand), which includes for instance variable selection in regression models, to nonparametric estimation, for which it provides a very powerful tool that allows adaptation under quite general circumstances. Our approach to model selection also provides a natural connection between the parametric and nonparametric points of view and copes naturally with the fact that a model is not necessarily true. The method is based on the penalization of a least squares criterion which can be viewed as a generalization of Mallows' C p . A large part of our efforts will be put on choosing properly the list of models and the penalty function for various estimation problems like classical variable selection or adaptive estimation for various types of l p -bodies. Introducing model selection from a nonasymptotic point of viewChoosing a proper parameter set is a difficult task in many estimation problems. A large one systematically leads to a large risk while a small one may result in the same consequence, due to unduly large bias. Both excessively complicated or oversimplified models should be avoided. The dilemna of the choice, between many possible models, of one which is adequate for the situation at hand, depending on both the unknown complexity of the true parameter to be estimated and the known amount of noise or number of observations, is often a nightmare for the statistician. The purpose of this paper is to provide a general methodology, namely model selection via penalization, for solving such problems within a unified Gaussian framework which covers many classical situations involving Gaussian variables. L. Birgé: UMR 7599 "Probabilités et modèles aléatoires

show abstract

“…Here we consider the ideal situation in which they are fixed; we concentrate on learning only. Assumptions (A1) and (A2) on the regression model (1) are supposed to be satisfied throughout the paper.…”

Section: Introductionmentioning

confidence: 99%

Aggregation and Sparsity Via ℓ1 Penalized Least Squares

2006

View full text Add to dashboard Cite

Abstract. This paper shows that near optimal rates of aggregation and adaptation to unknown sparsity can be simultaneously achieved via 1 penalized least squares in a nonparametric regression setting. The main tool is a novel oracle inequality on the sum between the empirical squared loss of the penalized least squares estimate and a term reflecting the sparsity of the unknown regression function.

show abstract

Model selection for regression on a random design

Cited by 86 publications

References 17 publications

Model selection via testing: an alternative to (penalized) maximum likelihood estimators

Model selection via testing: an alternative to (penalized) maximum likelihood estimators

Gaussian model selection

Aggregation and Sparsity Via ℓ1 Penalized Least Squares

Contact Info

Product

Resources

About