Predicting the Performance of Recommender Systems: An Information Theoretic Approach

User Modeling, Adaptation, and Personalization

Said

Vries

2014

Self Cite

Abstract. Recommender Systems need to deal with different types of users who represent their preferences in various ways. This difference in user behaviour has a deep impact on the final performance of the recommender system, where some users may receive either better or worse recommendations depending, mostly, on the quantity and the quality of the information the system knows about the user. Specifically, the inconsistencies of the user impose a lower bound on the error the system may achieve when predicting ratings for that particular user. In this work, we analyse how the consistency of user ratings (coherence) may predict the performance of recommendation methods. More specifically, our results show that our definition of coherence is correlated with the so-called magic barrier of recommender systems, and thus, it could be used to discriminate between easy users (those with a low magic barrier) and difficult ones (those with a high magic barrier). We report experiments where the rating prediction error for the more coherent users is lower than that of the less coherent ones. We further validate these results by using a public dataset, where the magic barrier is not available, in which we obtain similar performance improvements.

Section: A Measure Of User Coherence For Recommendationmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

The Magic Barrier of Recommender Systems – No Magic, Just Ratings

User Modeling, Adaptation, and Personalization

Said

Vries

2014

Self Cite

“…This work may also be observed from the perspective of performance prediction in Information Retrieval [7,16] and recommendation [3], where different functions have been proposed to predict the final performance of the query (in retrieval) or the target user (in recommendation). In our case, the features we have defined are assumed to be inherent to the components of the recommender system, and thus, they measure a global property of the algorithm and its corresponding similarity metric.…”

Section: Related Workmentioning

confidence: 99%

Understanding Similarity Metrics in Neighbour-based Recommender Systems

Proceedings of the 2013 Conference on the Theory of Information Retrieval

Vries

2013

Self Cite

Neighbour-based collaborative filtering is a recommendation technique that provides meaningful and, usually, accurate recommendations. The method's success depends however critically upon the similarity metric used to find the most similar users (neighbours), the basis of the predictions made. In this paper, we explore twelve features that aim to explain why some user similarity metrics perform better than others. Specifically, we define two sets of features, a first one based on statistics computed over the distance distribution in the neighbourhood, and, a second one based on the nearest neighbour graph. Our experiments with a public dataset show that some of these features are able to correlate with the performance up to a 90%.

“…In particular, under some conditions such as when positive correlation is found between the predictor and the recommender performance. Currently, we have found that our predictors are able to obtain strong (around 0.5) positive correlation with some recommenders [6]. We believe this result would allow for proposing a framework in which decisions about when and how predictors should be used for ensemble recommenders can be taken.…”

Section: Dynamic Ensemble Recommendationmentioning

confidence: 69%

“…This adaptation, however, is not straightforward, since RS rank and recommend items without an explicit user query, using other inputs instead. We explore different formulations of the user clarity under different models in [6] and [8]. Different assumptions for the random variables derive different models, all of them grounded on previous probability formulations in RS such as [15].…”

Section: Predicting Performance In Recommender Systemsmentioning

confidence: 99%

Predicting performance in recommender systems

Proceedings of the Fifth ACM Conference on Recommender Systems

2011

Performance prediction has gained growing attention in the Information Retrieval field since the late nineties and has become an established research topic in the field. Our work restates the problem in the area of Recommender Systems, where it has barely been researched so far, despite being an appealing problem, as it enables an array of strategies for deciding when to deliver or hold back recommendations based on their foreseen accuracy. We investigate the adaptation and definition of different performance predictors based on the available user and item features. The properties of the predictor are empirically studied by checking the correlation of the predictor output with a performance measure. Then, we propose to introduce the performance predictor in a recommender system to produce a dynamic strategy. Depending on how the predictor is introduced we analyze two different problems: dynamic neighbor weighting in collaborative filtering and dynamic weighting of ensemble recommenders.