A second-order method for strongly convex $$\ell _1$$ ℓ 1 -regularization problems

Fountoulakis, Kimon; Gondzio, Jacek

doi:10.1007/s10107-015-0875-4

Cited by 52 publications

(65 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Properties of the pseudo-Huber function and its application to compressed sensing problems are described in [16]. The objective function of the resulting convex optimization problem is…”

mentioning

confidence: 99%

Interior-point solver for convex separable block-angular problems

Castro

2015

Optimization Methods and Software

View full text Add to dashboard Cite

Constraints matrices with block-angular structures are pervasive in Optimization. Interior-point methods have shown to be competitive for these structured problems by exploiting the linear algebra. One of these approaches solved the normal equations using sparse Cholesky factorizations for the block constraints, and a preconditioned conjugate gradient (PCG) for the linking constraints. The preconditioner is based on a power series expansion which approximates the inverse of the matrix of the linking constraints system. In this work we present an efficient solver based on this algorithm. Some of its features are: it solves linearly constrained convex separable problems (linear, quadratic or nonlinear); both Newton and second-order predictor-corrector directions can be used, either with the Cholesky+PCG scheme or with a Cholesky factorization of normal equations; the preconditioner may include any number of terms of the power series; for any number of these terms, it estimates the spectral radius of the matrix in the power series (which is instrumental for the quality of the preconditioner). The solver has been hooked to SML, a structure-conveying modelling language based on the popular AMPL modeling language. Computational results are reported for some large and/or difficult instances in the literature: (1) multicommodity flow problems; (2) minimum congestion problems; (3) statistical data protection problems using ℓ 1 and ℓ 2 distances (which are linear and quadratic problems, respectively), and the pseudo-Huber function, a nonlinear approximation to ℓ 1 which improves the preconditioner. In the largest instances, of up to 25 millions of variables and 300000 constraints, this approach is from two to three orders of magnitude faster than state-of-the-art linear and quadratic optimization solvers.

show abstract

“…Properties of the pseudo-Huber function and its application to compressed sensing problems are described in [16]. The objective function of the resulting convex optimization problem is…”

mentioning

confidence: 99%

Interior-point solver for convex separable block-angular problems

Castro

2015

Optimization Methods and Software

View full text Add to dashboard Cite

show abstract

“…Thus, we assume that CG initialized with zero vector is employed at each Inexact Newton iteration. We will make use of the following technical Lemma that it is proved in [17].…”

Section: Inexact Subsampled Newton Methodsmentioning

confidence: 99%

Subsampled inexact Newton methods for minimizing large sums of convex functions

Bellavia

Krejić

Jerinkic

2019

IMA Journal of Numerical Analysis

View full text Add to dashboard Cite

This paper deals with the minimization of large sum of convex functions by Inexact Newton (IN) methods employing subsampled functions, gradients and Hessian approximations. The Conjugate Gradient method is used to compute the inexact Newton step and global convergence is enforced by a nonmonotone line search procedure. The aim is to obtain methods with affordable costs and fast convergence. Assuming strongly convex functions, R-linear convergence and worst-case iteration complexity of the procedure are investigated when functions and gradients are approximated with increasing accuracy. A set of rules for the forcing parameters and subsample Hessian sizes are derived that ensure local q-linear/superlinear convergence of the proposed method. The random choice of the Hessian subsample is also considered and convergence in the mean square, both for finite and infinite sums of functions, is proved. Finally, global convergence with asymptotic R-linear rate of IN methods is extended to the case of sum of convex function and strongly convex objective function. Numerical results on well known binary classification problems are also given. Adaptive strategies for selecting forcing terms and Hessian subsample size, streaming out of the theoretical analysis, are employed and the numerical results showed that they yield effective IN methods.

show abstract

“…Because all local minimizers of the twicedifferentiable functions f n are global minimizers from [Le and White, 2017, Theorem 10], we can conclude that all corresponding minimizers of f are global minimizers. We use the pseudo-Huber loss [Fountoulakis and Gondzio, 2013], which is twice-differentiable approximation to the absolute value: |x| µ = µ 2 + x 2 − µ. Let θ = (Φ, B, w).…”

Section: Local Minima Are Global Minimamentioning

confidence: 99%

Learning Sparse Representations in Reinforcement Learning with Sparse Coding

Kumaraswamy

White

2017

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

A variety of representation learning approaches have been investigated for reinforcement learning; much less attention, however, has been given to investigating the utility of sparse coding. Outside of reinforcement learning, sparse coding representations have been widely used, with non-convex objectives that result in discriminative representations. In this work, we develop a supervised sparse coding objective for policy evaluation. Despite the non-convexity of this objective, we prove that all local minima are global minima, making the approach amenable to simple optimization strategies. We empirically show that it is key to use a supervised objective, rather than the more straightforward unsupervised sparse coding approach. We compare the learned representations to a canonical fixed sparse representation, called tile-coding, demonstrating that the sparse coding representation outperforms a wide variety of tilecoding representations.

show abstract

A second-order method for strongly convex $$\ell _1$$ ℓ 1 -regularization problems

Cited by 52 publications

References 25 publications

Interior-point solver for convex separable block-angular problems

Interior-point solver for convex separable block-angular problems

Subsampled inexact Newton methods for minimizing large sums of convex functions

Learning Sparse Representations in Reinforcement Learning with Sparse Coding

Contact Info

Product

Resources

About