Degrees of freedom and model search

Tibshirani, Ryan J.

doi:10.5705/ss.2014.147

Cited by 31 publications

(62 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The reader is also referred to the recent paper (Tibshirani, 2014) where similar ideas have been independently developed to study the equivalent degrees of freedom of best subset selection.…”

Section: Estimation Of Hyperparameters and The Excess Of Degrees Of Fmentioning

confidence: 97%

See 1 more Smart Citation

Tuning complexity in regularized kernel-based regression and linear system identification: The robustness of the marginal likelihood estimator

Pillonetto

Chiuso

2015

Automatica

View full text Add to dashboard Cite

Section: Estimation Of Hyperparameters and The Excess Of Degrees Of Fmentioning

confidence: 97%

“…underestimates) the mean squared error E∥x−μ(x)∥ 2 . Extending the definitions in Hastie et al (2001) and Tibshirani (2014) to the case in which the noise variance Σ is not a scaled identity, we define the matricial degrees of freedom…”

Section: Optimism and Degrees Of Freedommentioning

confidence: 99%

Tuning complexity in regularized kernel-based regression and linear system identification: The robustness of the marginal likelihood estimator

Pillonetto

Chiuso

2015

Automatica

View full text Add to dashboard Cite

“…First, the additional tuning parameters increase the model space and thus the "degrees of freedom." Degrees of freedom relate directly to over-optimism (Tibshirani 2015). Traditionally one thinks of over-optimism as the difference between a model's performance on future data and its training error.…”

Section: Simulation Studiesmentioning

confidence: 99%

Gradient-based Regularization Parameter Selection for Problems With Nonsmooth Penalty Functions

Feng

Simon

2018

Journal of Computational and Graphical Statistics

View full text Add to dashboard Cite

In high-dimensional and/or non-parametric regression problems, regularization (or penalization) is used to control model complexity and induce desired structure. Each penalty has a weight parameter that indicates how strongly the structure corresponding to that penalty should be enforced. Typically the parameters are chosen to minimize the error on a separate validation set using a simple grid search or a gradient-free optimization method. It is more efficient to tune parameters if the gradient can be determined, but this is often difficult for problems with non-smooth penalty functions. Here we show that for many penalized regression problems, the validation loss is actually smooth almost-everywhere with respect to the penalty parameters. We can therefore apply a modified gradient descent algorithm to tune parameters. Through simulation studies on example regression problems, we find that increasing the number of penalty parameters and tuning them using our method can decrease the generalization error.

show abstract

“…Whenθ s is the linear regression estimator onto a set of predictor variables indexed by the parameter s, the rule in (11) encompasses model selection via C p minimization, which is a classical topic in statistics. In general, tuning parameter selection via SURE minimization has been widely advocated by authors across various problem settings, e.g., Donoho and Johnstone (1995); Johnstone (1999); Zou et al (2007); Zou and Yuan (2008); Tibshirani andTaylor (2011, 2012); Candes et al (2013); Ulfarsson and Solo (2013a,b); Chen et al (2015), just to name a few.…”

Section: Parameter Tuning Via Surementioning

confidence: 99%

“…Soft-thresholding estimators, like the shrinkage estimators of Section 3.1, have been studied extensively in the statistical literature; some key references that study risk properties of soft-thresholding estimators are Donoho and Johnstone (1994Johnstone ( , 1995Johnstone ( , 1998, and Chapters 8 and 9 of Johnstone (2015) give a thorough summary. The extension of Stein's formula from Tibshirani (2015), as given in (52), can be used to prove that the excess degrees of freedom of the SURE-tuned soft-thresholding estimator is nonnegative.…”

Section: Soft-thresholding Estimatorsmentioning

confidence: 99%

Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?

Tibshirani

2018

Journal of the American Statistical Association

Self Cite

View full text Add to dashboard Cite

Nearly all estimators in statistical prediction come with an associated tuning parameter, in one way or another. Common practice, given data, is to choose the tuning parameter value that minimizes a constructed estimate of the prediction error of the estimator; we focus on Stein's unbiased risk estimator, or SURE (Stein, 1981;Efron, 1986), which forms an unbiased estimate of the prediction error by augmenting the observed training error with an estimate of the degrees of freedom of the estimator. Parameter tuning via SURE minimization has been advocated by many authors, in a wide variety of problem settings, and in general, it is natural to ask: what is the prediction error of the SURE-tuned estimator? An obvious strategy would be simply use the apparent error estimate as reported by SURE, i.e., the value of the SURE criterion at its minimum, to estimate the prediction error of the SURE-tuned estimator. But this is no longer unbiased; in fact, we would expect that the minimum of the SURE criterion is systematically biased downwards for the true prediction error. In this paper, we define the excess optimism to be the amount of this downward bias in the SURE minimum. We argue that the following two properties motivate the study of excess optimism: (i) an unbiased estimate of excess optimism, added to the SURE criterion at its minimum, gives an unbiased estimate of the prediction error of the SURE-tuned estimator; (ii) excess optimism serves as an upper bound on the excess risk, i.e., the difference between the risk of the SURE-tuned estimator and the oracle risk (where the oracle uses the best fixed tuning parameter choice). We study excess optimism in two common settings: the families of shrinkage and subset regression estimators. Our main results include a James-Stein-like property of SURE-tuned shrinkage estimation, which is shown to dominate the MLE, and both upper and lower bounds on excess optimism for SURE-tuned subset regression; when the collection of subsets here is nested, our bounds are particularly tight, and reveal that in the case of no signal, the excess optimism is always in between 0 and 10 degrees of freedom, no matter how many models are being selected from. We also describe a bootstrap method for estimating excess optimism, and outline some extensions of our framework beyond the standard homoskedastic, squared error model that we consider throughout majority of the paper.

show abstract

Degrees of freedom and model search

Cited by 31 publications

References 28 publications

Tuning complexity in regularized kernel-based regression and linear system identification: The robustness of the marginal likelihood estimator

Tuning complexity in regularized kernel-based regression and linear system identification: The robustness of the marginal likelihood estimator

Gradient-based Regularization Parameter Selection for Problems With Nonsmooth Penalty Functions

Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?

Contact Info

Product

Resources

About