Algorithms for Difference-of-Convex (DC) Programs Based on Difference-of-Moreau-Envelopes Smoothing

Sun, Kaizhao; Sun, Xu Andy

doi:10.48550/arxiv.2104.01470

Cited by 4 publications

(12 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They develop an efficient approximated gradient descent method based on the Moreau envelope smoothing technique, inspired by recent advances in non-smooth DC optimization [137]. To increase the efficiency of large data processing, they use an efficient stochastic block coordinate update for solving each sub-problem inexactly.…”

Section: Deep Partial Auc Maximizationmentioning

confidence: 99%

AUC Maximization in the Era of Big Data and AI: A Survey

Yang¹,

Ying²

2022

Preprint

View full text Add to dashboard Cite

Area under the ROC curve, a.k.a. AUC, is a measure of choice for assessing the performance of a classifier for imbalanced data. AUC maximization refers to a learning paradigm that learns a predictive model by directly maximizing its AUC score. It has been studied for more than two decades dating back to late 90s and a huge amount of work has been devoted to AUC maximization since then. Recently, stochastic AUC maximization for big data and deep AUC maximization for deep learning have received increasing attention and yielded great impact for solving real-world problems. However, to the best our knowledge there is no comprehensive survey of related works for AUC maximization. This paper aims to address the gap by reviewing the literature in the past two decades. We not only give a holistic view of the literature but also present detailed explanations and comparisons of different papers from formulations to algorithms and theoretical guarantees. We also identify and discuss remaining and emerging issues for deep AUC maximization, and provide suggestions on topics for future work.

show abstract

Section: Deep Partial Auc Maximizationmentioning

confidence: 99%

AUC Maximization in the Era of Big Data and AI: A Survey

Yang¹,

Ying²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Due to non-smoothness, most existing DC optimization algorithms cannot be applied to our formulation. Motivated by Sun and Sun (2021), we approximate the two non-smooth convex components in the DC program by their Moreau envelopes and obtained a smooth approximation of the problem, which will be solved using the gradient descent method. Since the gradient of the smooth problem cannot be calculated explicitly, we approximated the gradient by solving the two proximal-point subproblems defined by each convex component using the stochastic block coordinate descent (SBCD) method.…”

Section: Introductionmentioning

confidence: 99%

Large-scale Optimization of Partial AUC in a Range of False Positive Rates

Yao¹,

Lin²,

Yang³

2022

Preprint

View full text Add to dashboard Cite

The area under the ROC curve (AUC) is one of the most widely used performance measures for classification models in machine learning. However, it summarizes the true positive rates (TPRs) over all false positive rates (FPRs) in the ROC space, which may include the FPRs with no practical relevance in some applications. The partial AUC, as a generalization of the AUC, summarizes only the TPRs over a specific range of the FPRs and is thus a more suitable performance measure in many real-world situations. Although partial AUC optimization in a range of FPRs had been studied, existing algorithms are not scalable to big data and not applicable to deep learning. To address this challenge, we cast the problem into a non-smooth difference-of-convex (DC) program for any smooth predictive functions (e.g., deep neural networks), which allowed us to develop an efficient approximated gradient descent method based on the Moreau envelope smoothing technique, inspired by recent advances in non-smooth DC optimization. To increase the efficiency of large data processing, we used an efficient stochastic block coordinate update in our algorithm. Our proposed algorithm can also be used to minimize the sum of ranked range loss, which also lacks efficient solvers. We established a complexity of Õ(1/ǫ 6 ) for finding a nearly ǫ-critical solution. Finally, we numerically demonstrated the effectiveness of our proposed algorithms for both partial AUC maximization and sum of ranked range loss minimization.

show abstract

“…They prove convergence to a socalled D-stationary point under some mild conditions; see [37] for the definition of D-stationarity, as well as [33] for more recent extensions of this work. The interested reader can find other variants of DCA in [19,32,33,40].…”

mentioning

confidence: 99%

On the rate of convergence of the Difference-of-Convex Algorithm (DCA)

Abbaszadehpeivasti¹,

Klerk²,

Zamani³

2021

Preprint

View full text Add to dashboard Cite

In this paper, we study the convergence rate of the DCA (Difference-of-Convex Algorithm), also known as the convex-concave procedure. The DCA is a popular algorithm for difference-of-convex (DC) problems, and known to converge to a stationary point under some assumptions. We derive a worst-case convergence rate of O(1/ √ N ) after N iterations of the objective gradient norm for certain classes of unconstrained DC problems. For constrained DC problems with convex feasible sets, we obtain a O(1/N ) convergence rate (in a well-defined sense). We give an example which shows the order of convergence cannot be improved for a certain class of DC functions. In addition, we obtain the same convergence rate for the DCA with regularization. Our results complement recent convergence rate results from the literature where it is assumed that the objective function satisfies the Lojasiewicz gradient inequality at stationary points. In particular, we do not make this assumption.

show abstract

Algorithms for Difference-of-Convex (DC) Programs Based on Difference-of-Moreau-Envelopes Smoothing

Cited by 4 publications

References 47 publications

AUC Maximization in the Era of Big Data and AI: A Survey

AUC Maximization in the Era of Big Data and AI: A Survey

Large-scale Optimization of Partial AUC in a Range of False Positive Rates

On the rate of convergence of the Difference-of-Convex Algorithm (DCA)

Contact Info

Product

Resources

About