Stochastic Optimization of Areas UnderPrecision-Recall Curves with Provable Convergence

Abstract

Areas under ROC (AUROC) and precision-recall curves (AUPRC) are common metrics for evaluating classification performance for imbalanced problems. Compared with AUROC, AUPRC is a more appropriate metric for highly imbalanced datasets. While direct optimization of AUROC has been studied extensively, optimization of AUPRC has been rarely explored. In this work, we propose a principled technical method to optimize AUPRC for deep learning. Our approach is based on maximizing the averaged precision (AP), which is an unbiased point estimator of AUPRC. We show that the surrogate loss function for AP is highly non-convex and more complicated than that of AUROC. We cast the objective into a sum of dependent compositional functions with inner functions dependent on random variables of the outer level. We propose efficient adaptive and non-adaptive stochastic algorithms with provable convergence guarantee under mild conditions by using recent advances in stochastic compositional optimization. Extensive experimental results on graph and image datasets demonstrate that our proposed method outperforms prior methods on imbalanced problems. To the best of our knowledge, our work represents the first attempt to optimize AUPRC with provable convergence. 1 1. * denotes equal contribution.

Momentum Accelerates the Convergence of Stochastic AUPRC Maximization

“…On the other hand, directly optimizing AUPRC is generally intractable due to the involved complicated integral operation. To mitigate this issue, most of the existing works seek to optimize certain estimator of AUPRC [4,42,43]. In this paper, we focus on maximizing average precision (AP), which is one of the most commonly used estimators in practice for the purpose of maximizing AUPRC.…”

“…Although a few studies have tried to optimize AP for AUPRC optimization [4,5,6,24,39,43,45], most of them are heuristic driven and do not provide any convergence guarantee. Recently, Qi et al [42] made a breakthrough towards optimizing a differentiable surrogate loss of AP with provable convergence guarantee. They cast the objective as a sum of non-convex compositional functions, and propose a principled stochastic method named SOAP for solving the special optimization problem.…”

“…The first is similar to the one proposed in [42]. However, an improved rate of this scheme is difficult to establish unless the sampled positive data include all positive examples due to a subtle randomness issue.…”

“…• We propose and analyze a family of adaptive algorithms by using different adaptive step sizes. We establish the same order of iteration complexity by employing the [42] Yes AMSGrad O(1/ǫ 5 ) SOAP (Adam-style) [42] No Adam second scheme mentioned above for updating the biased estimators of the ranking functions. Although Qi et al [42] also considered adaptive algorithms, their analysis only applies to the AMSGrad-style adaptive size, and suffers from the O(1/ǫ 5 ) complexity.…”

“…Although a few methods have been proposed for maximizing AUPRC, stochastic optimization of AUPRC with convergence guarantee remains an undeveloped territory. A recent work [42] has proposed a promising approach towards AUPRC based on maximizing a surrogate loss for the average precision, and proved an O(1/ǫ 5 ) complexity for finding an ǫ-stationary solution of the non-convex objective. In this paper, we further improve the stochastic optimization of AURPC by (i) developing novel stochastic momentum methods with a better iteration complexity of O(1/ǫ 4 ) for finding an ǫ-stationary solution; and (ii) designing a novel family of stochastic adaptive methods with the same iteration complexity of O(1/ǫ 4 ), which enjoy faster convergence in practice.…”