Sparse recovery via differential inclusions

Abstract

In this paper, we recover sparse signals from their noisy linear measurements by solving nonlinear differential inclusions, which is based on the notion of inverse scale space (ISS) developed in applied mathematics. Our goal here is to bring this idea to address a challenging problem in statistics, i.e. finding the oracle estimator which is unbiased and sign-consistent using dynamics. We call our dynamics Bregman ISS and Linearized Bregman ISS. A well-known shortcoming of LASSO and any convex regularization approaches lies in the bias of estimators. However, we show that under proper conditions, there exists a bias-free and sign-consistent point on the solution paths of such dynamics, which corresponds to a signal that is the unbiased estimate of the true signal and whose entries have the same signs as those of the true signs, i.e. the oracle estimator. Therefore, their solution paths are regularization paths better than the LASSO regularization path, since the points on the latter path are biased when sign-consistency is reached. We also show how to efficiently compute their solution paths in both continuous and discretized settings: the full solution paths can be exactly computed piece by piece, and a discretization leads to Linearized Bregman iteration, which is a simple iterative thresholding rule and easy to parallelize. Theoretical guarantees such as sign-consistency and minimax optimal l2-error bounds are established in both continuous and discrete settings for specific points on the paths. Early-stopping rules for identifying these points are given. The key treatment relies on the development of differential inequalities for differential inclusions and their discretizations, which extends the previous results and leads to exponentially fast recovering of sparse signals before selecting wrong ones.

Adaptive multi-penalty regularization based on a generalized Lasso path

“…Then there exists an index j ∈ I τ and σ j ∈ {±1}, as well as a sequence β k →β such that sgn(u β k ,α1,j ) = sgn(u β k ,α2,j ) = σ j for all k and some α l ≤ α 1 < α 2 ≤ α u . As a consequence, the KKT conditions (5) imply that…”

“…Indeed, provided with the signal's support, the signal entries can be easily recovered with optimal statistical rate [3]. Therefore, support recovery has been a topic of active and fruitful research in the last years [4,5,6]. One typically considers linear observation model problems of the form (1) Au…”

Uniform-in-time weak error analysis for stochastic gradient descent algorithms via diffusion approximation

“…The dynamics of discrete, iterative numerical algorithms can often be better understood from their continuous time limit, typically described by ordinary differential equations. This perspective has been proven fruitful in the analysis of many deterministic optimization algorithms [30,31,32,33,34]. An analogy of this type of continuous-timelimit analysis for SGD algorithms is provided by the diffusion approximation [18,20]: in any finite time interval, the distribution of X n defined by the SGD dynamics (4) is close to the distribution of the solution of the following SDE at time t = nη:…”

From Social to Individuals: A Parsimonious Path of Multi-Level Models for Crowdsourced Preference Aggregation

“…Such a minimizer is unbiased when sign consistency is reached, hence is statistically more accurate than any convex regularized estimator such as LASSO. For more details, we refer the readers to see [33] and references therein.…”