The LASSO with Non-linear Measurements is Equivalent to One With Linear Measurements

Abstract

Consider estimating an unknown, but structured (e.g. sparse, low-rank, etc.), signal x 0 ∈ R n from a vector y ∈ R m of measurements of the form y i = g i (a i T x 0 ), where the a i 's are the rows of a known measurement matrix A, and, g(•) is a (potentially unknown) nonlinear and random link-function. Such measurement functions could arise in applications where the measurement device has nonlinearities and uncertainties. It could also arise by design, e.g., g i (x) = sign(x + z i ), corresponds to noisy 1-bit quantized measurements. Motivated by the classical work of Brillinger, and more recent work of Plan and Vershynin, we estimate x 0 via solving the Generalized-LASSO, i.e., x := arg min x y − Ax 0 2 + λf (x) for some regularization parameter λ > 0 and some (typically non-smooth) convex regularizer f (•) that promotes the structure of x 0 , e.g. ℓ 1 -norm, nuclear-norm, etc. While this approach seems to naively ignore the nonlinear function g(•), both Brillinger (in the non-constrained case) and Plan and Vershynin have shown that, when the entries of A are iid standard normal, this is a good estimator of x 0 up to a constant of proportionality µ, which only depends on g(•). In this work, we considerably strengthen these results by obtaining explicit expressions for x − µx 0 2 , for the regularized Generalized-LASSO, that are asymptotically precise when m and n grow large. A main result is that the estimation performance of the Generalized LASSO with non-linear measurements is asymptotically the same as one whose measurements are linear y i = µa i T x 0 + σz i , with µ = Eγg(γ) and σ 2 = E(g(γ) − µγ) 2 , and, γ standard normal. To the best of our knowledge, the derived expressions on the estimation performance are the first-known precise results in this context. One interesting consequence of our result is that the optimal quantizer of the measurements that minimizes the estimation error of the Generalized LASSO is the celebrated Lloyd-Max quantizer.

Fast and Reliable Parameter Estimation from Nonlinear Observations

“…Thus, the theorem above shows that the projected gradient updates converge at a linear rate to a small neighborhood around the "true" solution µθ * . The radius of this neighborhood decreases with an increase in the number of samples n. The size of this radius is near-optimal and comparable to recent results [19,22,28] where the estimate is obtained by solving the convex optimization problem in (1.1) which applies only when the regularization function R is convex. Indeed, the size of this radius scales like n 0 n w 2 which up to a small constant is exactly the same as the result one would get when the model is linear of the form y = µXθ * + w. This is perhaps unexpected as it demonstrates that our nonlinear model exactly behaves like a fictitious linear model with the same effective noise!…”

“…The plots in Figure 1a clearly show that PGD iterates applied to nonlinear observations converge quickly to an estimate which is of the same size as the effective noise induced by the nonlinearity. In this sense, PGD iterates converge quickly to a reliable solution which has exactly the same quality as the optimal results obtained in [22,28]. 3 Figure 1a also clearly demonstrates that the behavior of the PGD iterates applied to both models are essentially the same further corroborating the results of Theorem 3.1.…”

Near-Optimal Bounds for Binary Embeddings of Arbitrary Sets

“…In this work, we consider embedding a high-dimensional set of points K to the hamming cube in a lower dimension, which is known as binary embedding. Binary embedding is a natural problem arising from quantization of the measurements and is connected to 1-bit compressed sensing as well as locality sensitive hashing [1,3,14,20,22,25,27]. In particular, given a subset K of the unit sphere in R n , and a dimensionality reduction map A ∈ R m×n , we wish to ensure that 1 m sgn(Ax), sgn(Ay) H − ang(x, y) ≤ δ.…”

“…More recently in [12] Jacques studies a related problem in which samples Ax are uniformly quantized instead of being discretized to {+1, −1}. There is also a growing amount of literature related to binary embedding and one-bit compressed sensing [1,3,22,25]. In this work, we significantly improve the distortion dependence bounds for the binary embedding of arbitrary sets.…”

L1-Regularized Least Squares for Support Recovery of High Dimensional Single Index Models with Gaussian Designs

“…Neykov et al (2015) analyzed two algorithms based on Sliced Inverse Regression (Li 1991) under the assumption that X ∼ N (0, I p×p ), and demonstrated that they can uncover the support optimally in terms of the rescaled sample size. Plan & Vershynin (2015) and Thrampoulidis et al (2015) demonstrated that a constrained version of LASSO can be used to obtain an L 2 consistent estimator of β 0 . None of these procedures provide results on the performance of the LASSO algorithm in support recovery, which relates to L 2 consistency but is a fundamentally different theoretical aspect.…”