Statistical biases in Information Retrieval metrics for recommender systems

Abstract

There is an increasing consensus in the Recommender Systems community that the dominant error-based evaluation metrics are insufficient, and mostly inadequate, to properly assess the practical effectiveness of recommendations. Seeking to evaluate recommendation rankings—which largely determine the effective accuracy in matching user needs—rather than predicted rating values, Information Retrieval metrics have started to be applied for the evaluation of recommender systems. In this paper we analyse the main issues and potential divergences in the application of Information Retrieval methodologies to recommender system evaluation, and provide a systematic characterisation of experimental design alternatives for this adaptation. We lay out an experimental configuration framework upon which we identify and analyse specific statistical biases arising in the adaptation of Information Retrieval metrics to recommendation tasks, namely sparsity and popularity biases. These biases considerably distort the empirical measurements, hindering the interpretation and comparison of results across experiments. We develop a formal characterisation and analysis of the biases upon which we analyse their causes and main factors, as well as their impact on evaluation metrics under different experimental configurations, illustrating the theoretical findings with empirical evidence. We propose two experimental design approaches that effectively neutralise such biases to a large extent. We report experiments validating our proposed experimental variants, and comparing them to alternative approaches and metrics that have been defined in the literature with similar or related purposes.This work was partially supported by the national Spanish Government (grants nr. TIN2013-47090-C3-2 and TIN2016-80630-P). We wish to express our gratitude to the anonymous reviewers whose insightful and generous feedback guided us in producing an enhanced version of the paper beyond the amendmentof flaws and shortcomings

Unifying Explicit and Implicit Feedback for Rating Prediction and Ranking Recommendation Tasks

“…Φ D i cannot exist. On the other hand, previous studies [1,2,20] have emphasized the importance of modeling implicit feedback as part of the recommendation model. In the following section, we leverage the explicit model described in Equation (4) as a weakly supervised signal and augment it with the implicit feedback at the level of data pre-processing.…”

“…The task of rating prediction is measured by comparing the predicted ratings with the ground truth; metrics such as Root Mean Square Error (RMSE) are useful in this context. On the other hand, a ranking task is usually measured based on Information Retrieval (IR) metrics including Mean Reciprocal Rank (MRR) and normalized Discounted Cumulative Gain (nDCG) [1,2]. We evaluate the performance of the proposed model on both the rating prediction and ranking tasks.…”

“…We evaluate the performance of the proposed model on both the rating prediction and ranking tasks. Therefore, we rank items based on the actual rating score r from the explicit test subset (D test e ), and thereby calculate both RMSE and rank-based measures, namely MRR, nDCG and MAP, as the evaluation measures [1,2]. Significance testing of differences between ranking measure performances are measured using the paired t-test (p < 0.01) 5 .…”

“…In contrast, the goal of a ranking task is to provide the user with a personalized rank list of items (also known as top-k recommendation). Despite the logical relation between these two tasks, different models and techniques have been developed in the literature and in fact researchers often distinguish between the two broad categories when measuring recommendation effectiveness [22]: the rating prediction task is usually quantified in terms of the Root Mean Square Error [20]; while the ranking task effectiveness is measured using Information Retrieval (IR) metrics [1]. Despite the popularity of the ranking task in the field of recommendation systems, ranking can be considered to be a sub-problem of the rating prediction task [22], since accurate rating prediction models can enable the ranking of the items with the highest predicted ratings.…”

“…It is well-known in both academia [13,20] and industry [22] that explicit models are more effective for the rating prediction task while implicit models are well suited to the ranking task. Therefore, when evaluating a system based on the construction of a ranking over all items, modeling implicit feedback is crucial [1].…”