A Block Decomposition Algorithm for Sparse Optimization

Yuan, Ganzhao; Shen, Li; Zheng, Wei‐Shi

doi:10.1145/3394486.3403070

Cited by 7 publications

(16 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Its convergence and worst-case complexity are well investigated for different coordinate selection rules such as cyclic rule (Beck and Tetruashvili 2013), greedy rule (Hsieh and Dhillon 2011), and random rule (Lu and Xiao 2015;Richtárik and Takávc 2014). It has been extended to solve many nonconvex problems such as penalized regression (Breheny and Huang 2011;Deng and Lan 2020), eigenvalue complementarity problem (Patrascu and Necoara 2015), 0 norm minimization (Beck and Eldar 2013;Yuan, Shen, and Zheng 2020), resource allocation problem (Necoara 2013), leading eigenvector computation (Li, Lu, and Wang 2019), and sparse phase retrieval (Shechtman, Beck, and Eldar 2014).…”

Section: Introductionmentioning

confidence: 99%

Coordinate Descent Methods for DC Minimization

Yuan¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Difference-of-Convex (DC) minimization, referring to the problem of minimizing the difference of two convex functions, has been found rich applications in statistical learning and studied extensively for decades. However, existing methods are primarily based on multi-stage convex relaxation, only leading to weak optimality of critical points. This paper proposes a coordinate descent method for minimizing DC functions based on sequential nonconvex approximation. Our approach iteratively solves a nonconvex one-dimensional subproblem globally, and it is guaranteed to converge to a coordinate-wise stationary point. We prove that this new optimality condition is always stronger than the critical point condition and the directional point condition when the objective function is weakly convex. For comparisons, we also include a naive variant of coordinate descent methods based on sequential convex approximation in our study. When the objective function satisfies an additional regularity condition called sharpness, coordinate descent methods with an appropriate initialization converge linearly to the optimal solution set. Also, for many applications of interest, we show that the nonconvex one-dimensional subproblem can be computed exactly and efficiently using a breakpoint searching method. We present some discussions and extensions of our proposed method. Finally, we have conducted extensive experiments on several statistical learning tasks to show the superiority of our approach.

show abstract

Section: Introductionmentioning

confidence: 99%

Coordinate Descent Methods for DC Minimization

Yuan¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…It is proven that this condition is stronger than the optimality criterion based on IHT. The work of [26,25] proposes and analyzes a new block coordinate optimality condition for general sparse optimization. The block coordinate optimality condition is stronger than the coordinate-wise optimality condition since it includes the latter as a special case.…”

Section: Introductionmentioning

confidence: 99%

“…(i) They fail to solve general nonsmooth sparsity constrained problems. The block decomposition methods [26,25] can only be applied to solve smooth optimization problems. Both the subgradient IHT method and dual IHT method can only solve problems where the objective function is Lipschitz continuous but fail in solving general non-Lipschitz problems as the penalty method can do.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Nonsmooth Sparsity Constrained Optimization via Penalty Alternating Direction Methods

Yuan¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Nonsmooth sparsity constrained optimization captures a broad spectrum of applications in machine learning and computer vision. However, this problem is NP-hard in general. Existing solutions to this problem are limited since they fail to solve general nonsmooth problems, lack convergence analysis, or lead to weaker optimality conditions. This paper revisits the Penalty Alternating Direction Method (PADM) for nonsmooth sparsity constrained optimization problems. We consider two variants of the PADM, i.e., PADM based on Iterative Hard Thresholding (PADM-IHT) and PADM based on Block Coordinate Decomposition (PADM-BCD). We show that the PADM-BCD algorithm finds stronger stationary points of the optimization problem than previous methods. We also develop novel theories to analyze the convergence rate for both of the PADM-IHT and the PADM-BCD algorithms. Finally, numerical results demonstrate the superiority of PADM-BCD to existing sparse optimization algorithms.

show abstract

“…For the general form of Problem (1), matching pursuit methods encounter the same issue as GraSP. Later, coordinate-wise algorithms and block decomposition algorithms also were developed (Patrascu and Necoara 2015;Yuan, Shen, and Zheng 2019). However, they may either cycle indefinitely, if the minimization step has multiple solutions, or need to solve a subproblem globally using combinatorial search methods at each iteration, which may fail for very large sparsity k. Hence, iterative gradient-based HT methods have gained significant interest and become popular for nonconvex sparse learning.…”

Section: Introductionmentioning

confidence: 99%

An Effective Hard Thresholding Method Based on Stochastic Variance Reduction for Nonconvex Sparse Learning

Liang

Tong

Zhu

et al. 2020

AAAI

View full text Add to dashboard Cite

We propose a hard thresholding method based on stochastically controlled stochastic gradients (SCSG-HT) to solve a family of sparsity-constrained empirical risk minimization problems. The SCSG-HT uses batch gradients where batch size is pre-determined by the desirable precision tolerance rather than full gradients to reduce the variance in stochastic gradients. It also employs the geometric distribution to determine the number of loops per epoch. We prove that, similar to the latest methods based on stochastic gradient descent or stochastic variance reduction methods, SCSG-HT enjoys a linear convergence rate. However, SCSG-HT now has a strong guarantee to recover the optimal sparse estimator. The computational complexity of SCSG-HT is independent of sample size n when n is larger than 1/ε, which enhances the scalability to massive-scale problems. Empirical results demonstrate that SCSG-HT outperforms several competitors and decreases the objective value the most with the same computational costs.

show abstract

A Block Decomposition Algorithm for Sparse Optimization

Cited by 7 publications

References 37 publications

Coordinate Descent Methods for DC Minimization

Coordinate Descent Methods for DC Minimization

Nonsmooth Sparsity Constrained Optimization via Penalty Alternating Direction Methods

An Effective Hard Thresholding Method Based on Stochastic Variance Reduction for Nonconvex Sparse Learning

Contact Info

Product

Resources

About