On Degrees of Freedom of Projection Estimators With Applications to Multivariate Nonparametric Regression

Chen, Xi; Lin, Qihang; Sen, Bodhisattva

doi:10.1080/01621459.2018.1537917

Cited by 21 publications

(12 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…. , n onto the space of θ; see [34]. The above characterization can be seen as a consequence of the subgradient inequality for convex functions; see [119,Theorem 25.1,p.…”

Section: Computation Of the Lsementioning

confidence: 99%

See 1 more Smart Citation

Nonparametric Shape-Restricted Regression

Guntuboyina¹,

Sen²

2018

Statist. Sci.

Self Cite

View full text Add to dashboard Cite

We consider the problem of nonparametric regression under shape constraints. The main examples include isotonic regression (with respect to any partial order), unimodal/convex regression, additive shape-restricted regression, and constrained single index model. We review some of the theoretical properties of the least squares estimator (LSE) in these problems, emphasizing on the adaptive nature of the LSE. In particular, we study the behavior of the risk of the LSE, and its pointwise limiting distribution theory, with special emphasis to isotonic regression. We survey various methods for constructing pointwise confidence intervals around these shape-restricted functions. We also briefly discuss the computation of the LSE and indicate some open research problems and future directions.Observe that when d = 1, convexity is characterized by nondecreasing derivatives (subgradients). This observation can be used to generalize convexity to k-monotonicity (k ≥ 1): a real-valued function f is said to be k-monotone if its (k − 1)'th derivative is monotone; see e.g., [94,28]. For equi-spaced design points in R, this restriction constrains θ * to lie in the setwhere ∇ : R n → R n is given by ∇(θ) := (θ 2 − θ 1 , θ 3 − θ 2 , . . . , θ n − θ n−1 , 0) and ∇ k represents the k-times composition of ∇. Note that the case k = 1 and k = 2 correspond to isotonic and convex regression, respectively.Example 1.4 (Unimodal regression). In many applications f , the underlying regression function, is known to be unimodal; see e.g., [49,27] and the references therein. Let I m , 1 ≤ m ≤ n, denote the convex set of all unimodal vectors (first decreasing and then increasing) with mode at position m, i.e.,

show abstract

“…. , n onto the space of θ; see [34]. The above characterization can be seen as a consequence of the subgradient inequality for convex functions; see [119,Theorem 25.1,p.…”

Section: Computation Of the Lsementioning

confidence: 99%

“…Even in one-dimensional convex regression, as far as we are aware, whetherf n (0+) is a O p (1) random variable is not known; see [52] for some results onf n (0+) and its derivative (also see [8]). This has motivated the study of bounded/penalized shape-restricted LSEs; see e.g., [34,143,84].…”

Section: Some Open Problemsmentioning

confidence: 99%

Nonparametric Shape-Restricted Regression

Guntuboyina¹,

Sen²

2018

Statist. Sci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…, y n−1 , y n − µ). Similar regularization methods for isotonic regression have been studied by Chen et al (2015), Wu et al (2015) and Luss and Rosset (2017).…”

Section: Dealing With Inconsistency At Boundariesmentioning

confidence: 99%

Estimating piecewise monotone signals

Minami¹

2020

Electron. J. Statist.

View full text Add to dashboard Cite

We study the problem of estimating piecewise monotone vectors. This problem can be seen as a generalization of the isotonic regression that allows a small number of order-violating changepoints. We focus mainly on the performance of the nearly-isotonic regression proposed by Tibshirani et al. (2011). We derive risk bounds for the nearly-isotonic regression estimators that are adaptive to piecewise monotone signals. The estimator achieves a near minimax convergence rate over certain classes of piecewise monotone signals under a weak assumption. Furthermore, we present an algorithm that can be applied to the nearly-isotonic type estimators on general weighted graphs. The simulation results suggest that the nearlyisotonic regression performs as well as the ideal estimator that knows the true positions of changepoints. keywords: piecewise monotone function, isotonic regression, nearly-isotonic regression, adaptive risk bounds

show abstract

“…Whenθ s is the linear regression estimator onto a set of predictor variables indexed by the parameter s, the rule in (11) encompasses model selection via C p minimization, which is a classical topic in statistics. In general, tuning parameter selection via SURE minimization has been widely advocated by authors across various problem settings, e.g., Donoho and Johnstone (1995); Johnstone (1999); Zou et al (2007); Zou and Yuan (2008); Tibshirani andTaylor (2011, 2012); Candes et al (2013); Ulfarsson and Solo (2013a,b); Chen et al (2015), just to name a few.…”

Section: Parameter Tuning Via Surementioning

confidence: 99%

Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?

Tibshirani

2018

Journal of the American Statistical Association

View full text Add to dashboard Cite

Nearly all estimators in statistical prediction come with an associated tuning parameter, in one way or another. Common practice, given data, is to choose the tuning parameter value that minimizes a constructed estimate of the prediction error of the estimator; we focus on Stein's unbiased risk estimator, or SURE (Stein, 1981;Efron, 1986), which forms an unbiased estimate of the prediction error by augmenting the observed training error with an estimate of the degrees of freedom of the estimator. Parameter tuning via SURE minimization has been advocated by many authors, in a wide variety of problem settings, and in general, it is natural to ask: what is the prediction error of the SURE-tuned estimator? An obvious strategy would be simply use the apparent error estimate as reported by SURE, i.e., the value of the SURE criterion at its minimum, to estimate the prediction error of the SURE-tuned estimator. But this is no longer unbiased; in fact, we would expect that the minimum of the SURE criterion is systematically biased downwards for the true prediction error. In this paper, we define the excess optimism to be the amount of this downward bias in the SURE minimum. We argue that the following two properties motivate the study of excess optimism: (i) an unbiased estimate of excess optimism, added to the SURE criterion at its minimum, gives an unbiased estimate of the prediction error of the SURE-tuned estimator; (ii) excess optimism serves as an upper bound on the excess risk, i.e., the difference between the risk of the SURE-tuned estimator and the oracle risk (where the oracle uses the best fixed tuning parameter choice). We study excess optimism in two common settings: the families of shrinkage and subset regression estimators. Our main results include a James-Stein-like property of SURE-tuned shrinkage estimation, which is shown to dominate the MLE, and both upper and lower bounds on excess optimism for SURE-tuned subset regression; when the collection of subsets here is nested, our bounds are particularly tight, and reveal that in the case of no signal, the excess optimism is always in between 0 and 10 degrees of freedom, no matter how many models are being selected from. We also describe a bootstrap method for estimating excess optimism, and outline some extensions of our framework beyond the standard homoskedastic, squared error model that we consider throughout majority of the paper.

show abstract

On Degrees of Freedom of Projection Estimators With Applications to Multivariate Nonparametric Regression

Cited by 21 publications

References 47 publications

Nonparametric Shape-Restricted Regression

Nonparametric Shape-Restricted Regression

Estimating piecewise monotone signals

Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?

Contact Info

Product

Resources

About