Nonparametric Shape-Restricted Regression

Abstract

We consider the problem of nonparametric regression under shape constraints. The main examples include isotonic regression (with respect to any partial order), unimodal/convex regression, additive shape-restricted regression, and constrained single index model. We review some of the theoretical properties of the least squares estimator (LSE) in these problems, emphasizing on the adaptive nature of the LSE. In particular, we study the behavior of the risk of the LSE, and its pointwise limiting distribution theory, with special emphasis to isotonic regression. We survey various methods for constructing pointwise confidence intervals around these shape-restricted functions. We also briefly discuss the computation of the LSE and indicate some open research problems and future directions.Observe that when d = 1, convexity is characterized by nondecreasing derivatives (subgradients). This observation can be used to generalize convexity to k-monotonicity (k ≥ 1): a real-valued function f is said to be k-monotone if its (k − 1)'th derivative is monotone; see e.g., [94,28]. For equi-spaced design points in R, this restriction constrains θ * to lie in the setwhere ∇ : R n → R n is given by ∇(θ) := (θ 2 − θ 1 , θ 3 − θ 2 , . . . , θ n − θ n−1 , 0) and ∇ k represents the k-times composition of ∇. Note that the case k = 1 and k = 2 correspond to isotonic and convex regression, respectively.Example 1.4 (Unimodal regression). In many applications f , the underlying regression function, is known to be unimodal; see e.g., [49,27] and the references therein. Let I m , 1 ≤ m ≤ n, denote the convex set of all unimodal vectors (first decreasing and then increasing) with mode at position m, i.e.,

Adaptive risk bounds in univariate total variation denoising and trend filtering

“…V , θ * ) against log n for θ * as in (20). The least squares slope is close to −2/3 which suggests that the risk decays as n −2/3 instead of the faster rate given by Corollary 2.3.…”

“…We shall argue here via simulations that the minimum length condition in Corollary 2.3 cannot be removed. Suppose that θ * is given by θ * 1 = · · · = θ * n−1 = 0 and θ * n = 5 (20) and consider estimating θ * from an observation Y ∼ N n (θ * , I n ) (i.e., σ = 1) byθ (1) V (i.e., r = 1) with tuning parameter V = V (1) (θ * ) = 5. It is clear here that k 1 (θ * ) = 1.…”

The bias of isotonic regression

“…Recently, there has been renewed interest in isotonic regression as one of the most widely-used examples of regression under shape constraints (see e.g. [Chatterjee et al, 2015, de Leeuw et al, 2009, Guntuboyina and Sen, 2018, Han et al, 2017). Given the significance of isotonic regression, understanding its statistical properties is of fundamental importance.…”

Efficient Second-Order Shape-Constrained Function Fitting

“…For a more comprehensive survey of shape-constrained function-fitting problems and their applications, see [14, §1]. Motivated by these applications, the problems have been studied in statistics (as a form of nonparametric regression), investigating, e.g., their consistency as estimators and their rate of convergence [13,14, 4].While fast algorithms for isotonic-regression variants have been designed [27], both [22] and [3] list shape constraints beyond monotonicity as important challenges. For example, fitting (multidimensional) convex functions is mostly done via quadratic or linear programming solvers [24].…”

“…For a more comprehensive survey of shape-constrained function-fitting problems and their applications, see [14, §1]. Motivated by these applications, the problems have been studied in statistics (as a form of nonparametric regression), investigating, e.g., their consistency as estimators and their rate of convergence [13,14, 4].…”