A Fast Active Set Block Coordinate Descent Algorithm for $\ell_1$-Regularized Least Squares

Abstract

Abstract. The problem of finding sparse solutions to underdetermined systems of linear equations arises in several applications (e.g. signal and image processing, compressive sensing, statistical inference). A standard tool for dealing with sparse recovery is the ℓ 1 -regularized least-squares approach that has been recently attracting the attention of many researchers.In this paper, we describe an active set estimate (i.e. an estimate of the indices of the zero variables in the optimal solution) for the considered problem that tries to quickly identify as many active variables as possible at a given point, while guaranteeing that some approximate optimality conditions are satisfied. A relevant feature of the estimate is that it gives a significant reduction of the objective function when setting to zero all those variables estimated active. This enables to easily embed it into a given globally converging algorithmic framework.In particular, we include our estimate into a block coordinate descent algorithm for ℓ 1 -regularized least squares, analyze the convergence properties of this new active set method, and prove that its basic version converges with linear rate.Finally, we report some numerical results showing the effectiveness of the approach.Key words. ℓ 1 -regularized least squares, active set, sparse optimization AMS subject classifications. 65K05, 90C25, 90C061. Introduction. The problem of finding sparse solutions to large underdetermined linear systems of equations has received a lot of attention in the last decades. This is due to the fact that several real-world applications can be formulated as linear inverse problems. A standard approach is the so called ℓ 2 -ℓ 1 unconstrained optimization problem:

A second-order method for convex1-regularized optimization with active-set prediction

“…Any variable j ∈ U F for which this equality does not hold, is removed from the set U F and added to U A . The set A k is then updated according to (9) and a new trial step is recomputed by solving (10). We repeat this corrective cycle until all predictions are correct and the trial pointx k satisfies (11).…”

“…Email: nitishkeskar2012@u.northwestern.edu SpaRSA and FISTA [3,10,37], and proximal Newton methods that compute a step by minimizing a piecewise quadratic model of (1) using (for example) a coordinate descent iteration [5,16,19,23,26,29,32,38]. The proposed algorithm also differs from methods that solve (1) by reformulating it as a bound constrained problem [12,18,27,28,33,35], and from recent methods that are specifically designed for the case when f is a convex quadratic [9,30,34].…”

Hybrid Random/Deterministic Parallel Algorithms for Convex and Nonconvex Big Data Optimization

“…Term II: Let us rewrite term II as The desired result (14) follows readily combining (16), (17), and (18).…”

“…Our focus is on problems with a huge number of variables, as those that can be encountered, e.g., in machine learning, compressed sensing, data mining, tensor factorization and completion, network optimization, image processing, genomics, etc.. We refer the reader to [2]- [14] and the books [15], [16] as entry points to the literature.…”

Second-order orthant-based methods with enriched Hessian information for sparse $$\ell _1$$ ℓ 1 -optimization

“…In [4] and [6] these type of directions were considered in connection with Newton and semismooth Newton type updates, respectively. Active set strategies based on second order information have also been recently envisaged ( [31,34,40]). This paper targets the question whether some useful information can also be extracted from the special structure of the 1 -norm in order to design a second-order method.…”