Obtaining Calibrated Probabilities with Personalized Ranking Models

Abstract

For personalized ranking models, the well-calibrated probability of an item being preferred by a user has great practical value. While existing work shows promising results in image classification, probability calibration has not been much explored for personalized ranking. In this paper, we aim to estimate the calibrated probability of how likely a user will prefer an item. We investigate various parametric distributions and propose two parametric calibration methods, namely Gaussian calibration and Gamma calibration. Each proposed method can be seen as a post-processing function that maps the ranking scores of pre-trained models to well-calibrated preference probabilities, without affecting the recommendation performance. We also design the unbiased empirical risk minimization framework that guides the calibration methods to learning of true preference probability from the biased user-item interaction dataset. Extensive evaluations with various personalized ranking models on real-world datasets show that both the proposed calibration methods and the unbiased empirical risk minimization significantly improve the calibration performance.

Top-Personalized-K Recommendation

“…They often output the ranking score that can have any value of an unbounded real number [12,50,64], making it difficult to treat it as a probability. Furthermore, even when a model is trained to output probabilities [13,35], it has been demonstrated that these probabilities may not accurately reflect the true likelihood (i.e., model miscalibration) [11,29].…”

“…We adopt Platt scaling 𝑔 𝜙 (𝑠) = 𝜎 (𝑎𝑠 + 𝑏) [46], a generalized form of temperature scaling [11]. This calibration function has been deployed effectively for model calibration in computer vision 7 𝜋 perK = 𝜋 𝑘 max [10,40], natural language processing [9], and recommender system [29,30]. The key difference is that PerK instantiates the calibration function for each user, while previous calibration work [29] deploys one global calibration function covering all users.…”

“…The remaining challenge is obtaining accurate interaction probability with an arbitrary recommender model. The recommender models do not necessarily output the accurate interaction probability [11,29]. They often output unbounded ranking scores that cannot be treated as probabilities [12,50] or miscalibrated interaction probabilities that do not accurately reflect the true likelihood of user-item interactions [13,35].…”

“…To address this problem, we propose the use of calibrated interaction probability obtained through the user-wise calibration function (Sec 5.3). The calibration function maps the ranking scores of the recommender model to wellcalibrated interaction probabilities [29]. We adopt Platt scaling [46] and instantiate it for each user to consider the different distributions of the ranking score across users.…”

Uncertainty Calibration for Counterfactual Propensity Estimation in Recommendation

“…The uncertainty quantification can be characterized by using Bayesian methods, ensemble methods or calibration with binning and scaling [1]. The uncertainty of personalized ranking probabilities is learned by uncertainty calibration methods [23,15] and later applied in the online advertising systems [40,43].…”