Iterative Thresholding for Sparse Approximations

Abstract

ABSTRACT.Sparse signal expansions represent or approximate a signal using a small number of elements from a large collection of elementary waveforms. Finding the optimal sparse expansion is known to be NP hard in general and non-optimal strategies such as Matching Pursuit, Orthogonal Matching Pursuit, Basis Pursuit and Basis Pursuit De-noising are often called upon. These methods show good performance in practical situations, however, they do not operate on the ℓ0 penalised cost functions that are often at the heart of the problem. In this paper we study two iterative algorithms that are minimising the cost functions of interest. Furthermore, each iteration of these strategies has computational complexity similar to a Matching Pursuit iteration, making the methods applicable to many real world problems. However, the optimisation problem is non-convex and the strategies are only guaranteed to find local solutions, so good initialisation becomes paramount. We here study two approaches. The first approach uses the proposed algorithms to refine the solutions found with other methods, replacing the typically used conjugate gradient solver. The second strategy adapts the algorithms and we show on one example that this adaptation can be used to achieve results that lie between those obtained with Matching Pursuit and those found with Orthogonal Matching Pursuit, while retaining the computational complexity of the Matching Pursuit algorithm.

Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods

“…As mentioned in the introduction, these results offer a complementary view to the theoretical develoments of [11,12]. They also provide at the same time a very general convergence result which can be immediately generalized to compressive sensing problems involving semialgebraic or real-analytic nonlinear measurements.…”

“…The convergence results the authors obtained involve different assumptions on the linear operator A: they either assume that A < 1 [11,Theorem 3] or that A satisfies the restricted isometry property [12,Theorem 4]. Our results show that convergence actually occurs for any linear map so long as the sequence (x k ) k∈N is bounded.…”

“…As in [11], we proceed in the spirit of Tikhonov regularization for the least squares method. Fix a parameter λ > 0.…”

“…This kind of problem is central in compressive sensing [29]. In [11,12] this problem is tackled by using a "hard iterative thresholding" algorithm…”

Distributed compressed sensing of Hyperspectral images via blind source separation

“…The sparse approximation step consists in estimating S with a fixed b A ( b A being the current estimate of A) and is implemented via an Iterative Hard Thresholding (IHT) [9] where each of the constraints of (4) are imposed at each iteration just after the gradient descent step. The dictionary update step consists in estimating A with a fixed b S ( b S being the estimate of S obtained from the previous step), and is obtained by multiplying the pseudo-inverse of b SΦ T to the right side of Y , and then setting the non-positive values to zero so as to impose the non-negativity constraint on A.…”