Estimating and Evaluating the Uncertainty of Rating Predictions and Top-n Recommendations in Recommender Systems

Abstract

Uncertainty is a characteristic of every data-driven application, including recommender systems. The quantification of uncertainty can be key to increasing user trust in recommendations or choosing which recommendations should be accompanied by an explanation; and uncertainty estimates can be used to accomplish recommender tasks such as active learning and co-training. Many uncertainty estimators are available but, to date, the literature has lacked a comprehensive survey and a detailed comparison. In this paper, we fulfil these needs. We review the existing methods for uncertainty estimation and metrics for evaluating uncertainty estimates, while also proposing some estimation methods and evaluation metrics of our own. Using two datasets, we compare the methods using the evaluation metrics that we describe, and we discuss their strengths and potential issues. The goal of this work is to provide a foundation to the field of uncertainty estimation in recommender systems, on which further research can be built.

Exploiting Rating Prediction Certainty for Recommendation Formulation in Collaborative Filtering

“…The number of neighbors of a user is not affected by differences in rating practices, since the two similarity measures used in the paper effectively tackle these diversities. More specifically, the CS metric measures the angle between the rating vectors, not their magnitude [65,66], and the PCC measure subtracts the mean of each user's rating from the corresponding user's rating values [30]. Regarding the average rating of an item, this is computed over the ratings entered by all users.…”