Sparse Approximations with Interior Point Methods

Abstract

Large-scale optimization problems that seek sparse solutions have become ubiquitous. They are routinely solved with various specialized first-order methods. Although such methods are often fast, they usually struggle with not-so-well conditioned problems. In this paper, specialized variants of an interior point-proximal method of multipliers are proposed and analyzed for problems of this class. Computational experience on a variety of problems, namely, multi-period portfolio optimization, classification of data coming from functional Magnetic Resonance Imaging, restoration of images corrupted by Poisson noise, and classification via regularized logistic regression, provides substantial evidence that interior point methods, equipped with suitable linear algebra, can offer a noticeable advantage over first-order approaches.

An interior point-proximal method of multipliers for convex quadratic programming

“…Indeed, this can be done by appending some additional linear equality constraints in (P), making the ℓ 1 regularization separable (e.g. see [21,). Using Fenchel duality, we can easily verify (see Appendix A.1) that the dual of (P) can be written as…”

“…Now if the primal-dual pair (P)-(D) is feasible, it must remain feasible for any positive semi-definite D, since in this case (P) can be written as a convex quadratic problem by appending appropriate (necessarily feasible) linear equality and inequality constraints (e.g. as in [21,51]). Thus, Assumption 1 suffices to guarantee that the solution set of (P)-(D) is non-empty.…”

“…see [20,24,25,38,39,41,49,59]), or interior point methods (IPMs) applied to a reformulation of (P) (e.g. see [21,28,31,51]). Most globalized Newton-like approaches or proximal point variants studied in the literature are developed for composite programming problems in which either g(x) = 0 (e.g.…”

“…The resulting linear systems that one has to solve are better conditioned than their (possibly regularized) interior-point counterparts (e.g. see [8,21,28,31,71]), however, convergence is expected to be slower, as the method does not enjoy the polynomial worst-case complexity of interiorpoint methods. Nevertheless, these better conditioned linear systems can in certain cases allow one to achieve better computational and/or memory efficiency.…”

“…As a result, the associated linear systems solved within SSN have significantly smaller dimensions, compared to linear systems arising within interior point methods suitable for the solution of ℓ 1 -regularized convex quadratic problems (e.g. see [21,31,51]), potentially making the proposed approach a more attractive alternative for large-scale instances, such as those arising from the optimal control of PDEs. Finally, we propose an efficient warm-starting strategy based on a proximal alternating direction method of multipliers to further improve the efficiency of the approach at a low computational cost.…”