Estimating residual variance in nonparametric regression using least squares

Abstract

We propose a new estimator for the error variance in a nonparametric regression model. We estimate the error variance as the intercept in a simple linear regression model with squared differences of paired observations as the dependent variable and squared distances between the paired covariates as the regressor. For the special case of a one-dimensional domain with equally spaced design points, we show that our method reaches an asymptotic optimal rate which is not achieved by some existing methods. We conduct extensive simulations to evaluate finite-sample performance of our method and compare it with existing methods. Our method can be extended to nonparametric regression models with multivariate functions defined on arbitrary subsets of normed spaces, possibly observed on unequally spaced or clustered designed points.

On the Choice of Difference Sequence in a Unified Framework for Variance Estimation in Nonparametric Regression

“…To improve the literature, Tong and Wang (2005) have proposed a new direction for estimating the residual variance, inspired by the fact that the Rice estimator is always positively biased. Their linear regression method not only eliminated the estimation bias, but also reduced the estimation variance dramatically and hence achieved the asymptotically optimal rate of MSE for variance estimation.…”

“…If we fix m = 1 and allow r ≥ 2,σ 2 (r, m) results in the classical difference-based estimators including Gasser, Sroka and Jennen-Steinmetz (1986), Hall, Kay and Titterington (1990) and Dette, Munk and Wagner (1998). On the other side, if we fix r = 1 and allow m ≥ 2, thenσ 2 (r, m) results in the linear regression estimators in Tong and Wang (2005) and Tong, Ma and Wang (2013). From this point of view, the unified framework has greatly enriched the existing literature on the difference-based estimation in nonparametric regression.…”

“…Note that s k (r) in (8) can be represented as a combination of several lag-k Rice estimators. Hence, in spirit to the simulation-extrapolation method of Cook and Stefanski (1994) and Stefanski and Cook (1995), and the empirical-bias bandwidth selection method of Ruppert (1997), the unified estimator σ 2 (r, m) can be treated as a variance reduced estimator in comparison with the linear regression estimator in Tong and Wang (2005).…”

“…The main goal of the paper is to provide a smart solution for the very challenging difference sequence selection problem. To achieve this, we propose a unified framework for estimating σ 2 that combines the linear regression method in Tong and Wang (2005) with the higher-order difference estimators systematically. By this combination, the unified framework integrates the existing literature on the difference-based estimation and it has, but not limited to, the following major contributions: (i) the unified framework generates a very large family of estimators that includes most existing estimators as special cases; (ii) all existing difference-based estimators are shown to be suboptimal in the proposed family of estimators; and (iii) in the unified framework, the ordinary difference sequence can be widely used no matter if the sample size is small and/or the signal-to-noise ratio is large.…”

A simple bootstrap method for constructing nonparametric confidence bands for functions

“…There is an extensive literature on computing estimatorsσ 2 of the error variance σ 2 = var(ε); see, for example, Rice (1984), Buckley, Eagleson and Silverman (1988), Gasser, Sroka and Jennen-Steinmetz (1986), Stadtmüller (1987, 1993), Hall, Kay and Titterington (1990), Hall and Marron (1990), Seifert, Gasser and Wolf (1993), Neumann (1994), Müller and Zhao (1995), Dette, Munk and Wagner (1998), Fan and Yao (1998), Müller, Schick and Wefelmeyer (2003), Munk et al (2005), Tong and Wang (2005), Brown and Levine (2007), Cai, Levine andWang (2009), andMendez andLohr (2011). It includes residual-based estimators, which we introduce at (2.8) below, and methods based on differences and generalised differences.…”