Efficiency of Coordinate Descent Methods for Structured Nonconvex Optimization

Deng, Qi; Lan, Chenghao

doi:10.1007/978-3-030-67664-3_5

Cited by 2 publications

(2 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The algorithms mentioned above are deterministic and require processing the entire dataset per iteration. Stochastic algorithms that process only a small data sample per iteration have been studied (Mairal, 2013;Nitanda and Suzuki, 2017;Le Thi et al, 2017;Deng and Lan, 2020;He et al, 2021). However, they all assumed smoothness on at least one of the two convex components in the DC program.…”

Section: Related Workmentioning

confidence: 99%

Large-scale Optimization of Partial AUC in a Range of False Positive Rates

Yao¹,

Lin²,

Yang³

2022

Preprint

View full text Add to dashboard Cite

The area under the ROC curve (AUC) is one of the most widely used performance measures for classification models in machine learning. However, it summarizes the true positive rates (TPRs) over all false positive rates (FPRs) in the ROC space, which may include the FPRs with no practical relevance in some applications. The partial AUC, as a generalization of the AUC, summarizes only the TPRs over a specific range of the FPRs and is thus a more suitable performance measure in many real-world situations. Although partial AUC optimization in a range of FPRs had been studied, existing algorithms are not scalable to big data and not applicable to deep learning. To address this challenge, we cast the problem into a non-smooth difference-of-convex (DC) program for any smooth predictive functions (e.g., deep neural networks), which allowed us to develop an efficient approximated gradient descent method based on the Moreau envelope smoothing technique, inspired by recent advances in non-smooth DC optimization. To increase the efficiency of large data processing, we used an efficient stochastic block coordinate update in our algorithm. Our proposed algorithm can also be used to minimize the sum of ranked range loss, which also lacks efficient solvers. We established a complexity of Õ(1/ǫ 6 ) for finding a nearly ǫ-critical solution. Finally, we numerically demonstrated the effectiveness of our proposed algorithms for both partial AUC maximization and sum of ranked range loss minimization.

show abstract

Section: Related Workmentioning

confidence: 99%

Large-scale Optimization of Partial AUC in a Range of False Positive Rates

Yao¹,

Lin²,

Yang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Its convergence and worst-case complexity are well investigated for different coordinate selection rules such as cyclic rule (Beck and Tetruashvili 2013), greedy rule (Hsieh and Dhillon 2011), and random rule (Lu and Xiao 2015;Richtárik and Takávc 2014). It has been extended to solve many nonconvex problems such as penalized regression (Breheny and Huang 2011;Deng and Lan 2020), eigenvalue complementarity problem (Patrascu and Necoara 2015), 0 norm minimization (Beck and Eldar 2013;Yuan, Shen, and Zheng 2020), resource allocation problem (Necoara 2013), leading eigenvector computation (Li, Lu, and Wang 2019), and sparse phase retrieval (Shechtman, Beck, and Eldar 2014).…”

Section: Introductionmentioning

confidence: 99%

Coordinate Descent Methods for DC Minimization

Yuan¹

2021

Preprint

View full text Add to dashboard Cite

Difference-of-Convex (DC) minimization, referring to the problem of minimizing the difference of two convex functions, has been found rich applications in statistical learning and studied extensively for decades. However, existing methods are primarily based on multi-stage convex relaxation, only leading to weak optimality of critical points. This paper proposes a coordinate descent method for minimizing DC functions based on sequential nonconvex approximation. Our approach iteratively solves a nonconvex one-dimensional subproblem globally, and it is guaranteed to converge to a coordinate-wise stationary point. We prove that this new optimality condition is always stronger than the critical point condition and the directional point condition when the objective function is weakly convex. For comparisons, we also include a naive variant of coordinate descent methods based on sequential convex approximation in our study. When the objective function satisfies an additional regularity condition called sharpness, coordinate descent methods with an appropriate initialization converge linearly to the optimal solution set. Also, for many applications of interest, we show that the nonconvex one-dimensional subproblem can be computed exactly and efficiently using a breakpoint searching method. We present some discussions and extensions of our proposed method. Finally, we have conducted extensive experiments on several statistical learning tasks to show the superiority of our approach.

show abstract

Efficiency of Coordinate Descent Methods for Structured Nonconvex Optimization

Cited by 2 publications

References 21 publications

Large-scale Optimization of Partial AUC in a Range of False Positive Rates

Large-scale Optimization of Partial AUC in a Range of False Positive Rates

Coordinate Descent Methods for DC Minimization

Contact Info

Product

Resources

About