SparseNet: Coordinate Descent With Nonconvex Penalties

Abstract

We address the problem of sparse selection in linear models. A number of nonconvex penalties have been proposed in the literature for this purpose, along with a variety of convex-relaxation algorithms for finding good solutions. In this article we pursue a coordinate-descent approach for optimization, and study its convergence properties. We characterize the properties of penalties suitable for this approach, study their corresponding threshold functions, and describe a df-standardizing reparametrization that assists our pathwise algorithm. The MC+ penalty is ideally suited to this task, and we use it to demonstrate the performance of our algorithm. Certain technical derivations and experiments related to this article are included in the Supplementary Materials section.

Coordinate Descent Algorithm for Ramp Loss Linear Programming Support Vector Machines

“…Later, an extension of the coordinate gradient descent algorithm is proposed in [31] to deal with SVM problems of which the objectives consist of smooth functions and separable convex functions. The l 2 regularized SVMs with several non-convex losses are discussed in [24]. When it comes to Algorithm 2, we will adopt the statistic order of the updated coordinates, while specifically, we choose the uniform distribution.…”

Optimization of ℓp-regularized Linear Models via Coordinate Descent

“…p -"norm" was also mentioned in [6], but authors abandoned using coordinate descent procedure in this case, because they encountered some instability (caused by discontinuity in path of solutions, what is depicted in the Figure 1) and impossibility of converging to the global minimum (using Multi-stage Local Linear Approximation), even for some univariate case. Some non-convex regularizers were also studied in [7] and p quasi-norm was considered in [8].…”

Sparsest factor analysis for clustering variables: a matrix decomposition approach

“…is the resulting value of the negative log likelihood (2), and the degree of freedom df(ρ, γ), which is a function of ρ and γ, is obtained according to Mazumder, Friedman, & Hastie (2011) (Hirose & Yamamoto, 2014). However, the resulting combination is not guaranteed to give a solution with the desired Card(Λ).…”