Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?

Abstract

Nearly all estimators in statistical prediction come with an associated tuning parameter, in one way or another. Common practice, given data, is to choose the tuning parameter value that minimizes a constructed estimate of the prediction error of the estimator; we focus on Stein's unbiased risk estimator, or SURE (Stein, 1981;Efron, 1986), which forms an unbiased estimate of the prediction error by augmenting the observed training error with an estimate of the degrees of freedom of the estimator. Parameter tuning via SURE minimization has been advocated by many authors, in a wide variety of problem settings, and in general, it is natural to ask: what is the prediction error of the SURE-tuned estimator? An obvious strategy would be simply use the apparent error estimate as reported by SURE, i.e., the value of the SURE criterion at its minimum, to estimate the prediction error of the SURE-tuned estimator. But this is no longer unbiased; in fact, we would expect that the minimum of the SURE criterion is systematically biased downwards for the true prediction error. In this paper, we define the excess optimism to be the amount of this downward bias in the SURE minimum. We argue that the following two properties motivate the study of excess optimism: (i) an unbiased estimate of excess optimism, added to the SURE criterion at its minimum, gives an unbiased estimate of the prediction error of the SURE-tuned estimator; (ii) excess optimism serves as an upper bound on the excess risk, i.e., the difference between the risk of the SURE-tuned estimator and the oracle risk (where the oracle uses the best fixed tuning parameter choice). We study excess optimism in two common settings: the families of shrinkage and subset regression estimators. Our main results include a James-Stein-like property of SURE-tuned shrinkage estimation, which is shown to dominate the MLE, and both upper and lower bounds on excess optimism for SURE-tuned subset regression; when the collection of subsets here is nested, our bounds are particularly tight, and reveal that in the case of no signal, the excess optimism is always in between 0 and 10 degrees of freedom, no matter how many models are being selected from. We also describe a bootstrap method for estimating excess optimism, and outline some extensions of our framework beyond the standard homoskedastic, squared error model that we consider throughout majority of the paper.

Messy Data, Robust Inference? Navigating Obstacles to Inference with bigKRLS

“…Though kernels are theoretically well suited to high dimensions (Diaconis, Goel, and Holmes 2008; El Karoui 2010), large numbers of variables still create practical problems, both computational and interpretative. Statistically, recent theoretical advances in selective inference (Taylor and Tibshirani 2015), risk estimation for tuning parameters (Tibshirani and Rosset 2016), and subsampling (Boutsidis, Mahoney, and Drineas 2009; Gu, Jeon, and Lin 2013; Homrighausen and McDonald 2016) offer paths forward in high-dimensional space. In the meantime, our analysis suggests that KRLS can produce interpretable and theoretically useful estimates on a perennial topic of interest: the behavior of American voters.…”

A comment and erratum on "Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?"

“…We prove the theorem in Section 2.3, providing a bit of commentary here by discussing subset regression estimators, as in the paper [4,Sec. 4], then discussing more general smoothers and estimates [2].…”

“…Note that Z = Y − θ 0 and E[Z] = 0, and define the estimated Y i by Y i = [H s(Y ) (Y )] i . Then following [4], we see that if we define the excess degrees of freedom…”

“…Tibshirani and Rosset [4] study the excess risk that using s(Y ) in place of s 0 induces in the estimate θ s of θ 0 , defining the excess optimism…”

“…This a corrected version of [4,Equation (41)], as the referenced equation omits the second term. In this note, we provide bounds on this corrected edf quantity.…”