Optimizing Two-Way Partial AUC With an End-to-End Framework

Yang, Zhiyong; Xu, Qianqian; Bao, Shanhu; He, Yuan; Cao, Xiaochun; Huang, Qingming

doi:10.1109/tpami.2022.3185311

Cited by 13 publications

(9 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We conduct numerical experiments on both synthetic and real-world datasets to validate its utility and the fairness performance. One future direction is to consider fairness metrics involving group-level partial AUCs (Narasimhan and Agarwal 2017;Yang et al 2021b).…”

Section: Discussionmentioning

confidence: 99%

Minimax AUC Fairness: Efficient Algorithm with Provable Convergence

Yang

Lok

Varshney³

et al. 2023

AAAI

View full text Add to dashboard Cite

The use of machine learning models in consequential decision making often exacerbates societal inequity, in particular yielding disparate impact on members of marginalized groups defined by race and gender. The area under the ROC curve (AUC) is widely used to evaluate the performance of a scoring function in machine learning, but is studied in algorithmic fairness less than other performance metrics. Due to the pairwise nature of the AUC, defining an AUC-based group fairness metric is pairwise-dependent and may involve both intra-group and inter-group AUCs. Importantly, considering only one category of AUCs is not sufficient to mitigate unfairness in AUC optimization. In this paper, we propose a minimax learning and bias mitigation framework that incorporates both intra-group and inter-group AUCs while maintaining utility. Based on this Rawlsian framework, we design an efficient stochastic optimization algorithm and prove its convergence to the minimum group-level AUC. We conduct numerical experiments on both synthetic and real-world datasets to validate the effectiveness of the minimax framework and the proposed optimization algorithm.

show abstract

Section: Discussionmentioning

confidence: 99%

Minimax AUC Fairness: Efficient Algorithm with Provable Convergence

Yang

Lok

Varshney³

et al. 2023

AAAI

View full text Add to dashboard Cite

show abstract

“…Hence, they propose a new metric named Two-way Partial AUC (TPAUC), which pays attention to the upper-left head region under the ROC curve. Then, [40] first proposes an end-toend TPAUC optimization framework, which has a profound impact on subsequent work [41]. Nevertheless, TPAUC does not align with the Top-K ranking metrics in the recommendation.…”

Section: Partial Auc and Its Optimizationmentioning

confidence: 99%

“…Regarding the optimization of partial AUC, previous works [7,21,24,26] rely on full-batch optimization and the approximation of the Top (Bottom)-K ranking, leading to immeasurable biases and inefficiency. Recently, novel end-to-end mini-batch optimization frameworks have been proposed [40,42,44]. These methods can be extended to optimize our proposed LLPAUC metric.…”

Section: Partial Auc and Its Optimizationmentioning

confidence: 99%

Lower-Left Partial AUC: An Effective and Efficient Optimization Metric for Recommendation

Shi,

Wang,

Feng

et al. 2024

Proceedings of the ACM Web Conference 2024

View full text Add to dashboard Cite

Optimization metrics are crucial for building recommendation systems at scale. However, an effective and efficient metric for practical use remains elusive. While Top-K ranking metrics are the gold standard for optimization, they suffer from significant computational overhead. Alternatively, the more efficient accuracy and AUC metrics often fall short of capturing the true targets of recommendation tasks, leading to suboptimal performance. To overcome this dilemma, we propose a new optimization metric, Lower-Left Partial AUC (LLPAUC), which is computationally efficient like AUC but strongly correlates with Top-K ranking metrics. Compared to AUC, LLPAUC considers only the partial area under the ROC curve in the Lower-Left corner to push the optimization focus on Top-K. We provide theoretical validation of the correlation between LLPAUC and Top-K ranking metrics and demonstrate its robustness to noisy user feedback. We further design an efficient point-wise recommendation loss to maximize LLPAUC and evaluate it on three datasets, validating its effectiveness and robustness. The code is available at https://github.com/swt-user/LLPAUC.

show abstract

“…Nevertheless, these algorithms are not scalable and applicable to deep learning. More recently, [38] considers two-way partial AUC maximization and simplifies the optimizing problem for large scale optimization. [41] proposes new formulations of Partial AUC surrogate objectives using distributionally robust optimization (DRO).…”

Section: Partial Auc Maximizationmentioning

confidence: 99%

On the Theories Behind Hard Negative Sampling for Recommendation

Shi,

Chen,

Feng

et al. 2023

Preprint

View full text Add to dashboard Cite

Negative sampling has been heavily used to train recommender models on large-scale data, wherein sampling hard examples usually not only accelerates the convergence but also improves the model accuracy. Nevertheless, the reasons for the effectiveness of Hard Negative Sampling (HNS) have not been revealed yet. In this work, we fill the research gap by conducting thorough theoretical analyses on HNS. Firstly, we prove that employing HNS on the Bayesian Personalized Ranking (BPR) learner is equivalent to optimizing One-way Partial AUC (OPAUC). Concretely, the BPR equipped with Dynamic Negative Sampling (DNS) is an exact estimator, while with softmax-based sampling is a soft estimator. Secondly, we prove that OPAUC has a stronger connection with Top-𝐾 evaluation metrics than AUC and verify it with simulation experiments. These analyses establish the theoretical foundation of HNS in optimizing Top-𝐾 recommendation performance for the first time. On these bases, we offer two insightful guidelines for effective usage of HNS: 1) the sampling hardness should be controllable, e.g., via pre-defined hyper-parameters, to adapt to different Top-𝐾 metrics and datasets; 2) the smaller the 𝐾 we emphasize in Top-𝐾 evaluation metrics, the harder the negative samples we should draw. Extensive experiments on three real-world benchmarks verify the two guidelines. CCS CONCEPTS• Information systems → Recommender systems.

show abstract

Optimizing Two-Way Partial AUC With an End-to-End Framework

Cited by 13 publications

References 50 publications

Minimax AUC Fairness: Efficient Algorithm with Provable Convergence

Minimax AUC Fairness: Efficient Algorithm with Provable Convergence

Lower-Left Partial AUC: An Effective and Efficient Optimization Metric for Recommendation

On the Theories Behind Hard Negative Sampling for Recommendation

Contact Info

Product

Resources

About