A coordinate gradient descent method for ℓ 1-regularized convex minimization

Abstract

In applications such as signal processing and statistics, many problems involve finding sparse solutions to under-determined linear systems of equations. These problems can be formulated as a structured nonsmooth optimization problems, i.e., the problem of minimizing 1 -regularized linear least squares problems. In this paper, we propose a block coordinate gradient descent method (abbreviated as CGD) to solve the more general 1 -regularized convex minimization problems, i.e., the problem of minimizing an 1 -regularized convex smooth function. We establish a Q-linear convergence rate for our method when the coordinate block is chosen by a Gauss-Southwell-type rule to ensure sufficient descent. We propose efficient implementations of the CGD method and report numerical results for solving large-scale 1 -regularized linear least squares problems arising in compressed sensing and image deconvolution as well as large-scale 1 -regularized logistic regression problems for feature selection in data classification. Comparison with several state-of-the-art algorithms specifically designed for solving large-scale 1 -regularized linear least squares or logistic regression problems suggests that an efficiently implemented CGD method may outperform these algorithms despite the fact that the CGD method is not specifically designed just to solve these special classes of problems.

Block-Coordinate Gradient Descent Method for Linearly Constrained Nonsmooth Separable Optimization

“…This method approximates the smooth function by a quadratic function at the current iterate, applies block coordinate descent to generate a feasible descent direction, and then updates the current iterate by performing an inexact line search along the descent direction. Numerical performances in [25,31] suggest that the BCGD method can be effective in practice. We extend this method to solve (8) and, in particular, the covariance selection problems (1) with α = 0 and β = +∞, (5) with ρ ij > 0 for (i, j) ∈ V , and the dual problems (2), (7) with ρ ij > 0 for (i, j) ∈ V .…”

“…where the third inequality uses the self-adjoint positive definite property ofH k and the last inequality uses (31).…”

Recent Advances of Large-Scale Linear Classification

“…Some examples include Nyström mehtod [79], [80] and incomplete Cholesky factorization [81], [82]. Some works (e.g., [19]) consider approximations other than (39), but also lead to linear classification problems. A recent study [78] addresses more on training and testing linear SVM after obtaining the low-rank approximation.…”

“…• L1-regularized LR: Most methods solve the primal form, for example, an interior-point method (l1 logreg [37]), (block) coordinate descent methods (BBR [38] and CGD [39]), a quasi-Newton method (OWL-QN [40]), Newton-type methods (GLMNET [41] and LIBLINEAR [22]), and a Nesterov's method (SLEP [42]). Recently, an augmented Lagrangian method (DAL [43]) is proposed for solving the dual problem.…”

Generalized Sparse Learning of Linear Models Over the Complete Subgraph Feature Set

“…Our main idea is first to make block coordinate gradient descent [35], [36] feasible over all subgraph indicators. Then, by setting a small block size to update, Problem (2) with an intractably large number of variables becomes practically solvable.…”