User-Centric vs. System-Centric Evaluation of Recommender Systems

Abstract

Abstract. Recommender Systems (RSs) aim at helping users search large amounts of contents and identify more effectively the items (products or services) that are likely to be more useful or attractive. The quality of a RS can be defined from two perspectives: system-centric, in which quality measures (e.g., precision, recall) are evaluated using vast datasets of preferences and opinions on items previously collected from users that are not interacting with the RS under study; user-centric, in which user measures are collected from users interacting with the RS under study. Prior research in e-commerce has provided some empirical evidence that system-centric and user-centric quality methods may lead to inconsistent results, e.g., RSs that were "best" according to systemcentric measures were not the top ones according to user-centric measures. The paper investigates if a similar mismatch also exists in the domain of e-tourism. We discuss two studies that have adopted a system-centric approach using data from 210000 users, and a user-centric approach involving 240 users interacting with an online hotel booking service. In both studies, we considered four RSs that employ an implicit user preference elicitation technique and different baseline and state-of-the-art recommendation algorithms. In these four experimental conditions, we compared system-centric quality measures against user-centric evaluation results. System-centric quality measures were consistent with usercentric measures, in contrast with past studies in e-commerce. This pinpoints that the relationship between the two kinds of metrics may depend on the business sector, is more complex that we may expect, and is a challenging issues that deserves further research.

User interface patterns in recommendation-empowered content intensive multimedia applications

“…Also other works [13,23,24,27,61,77] have pinpointed that system-centric quality might not always correlate with user-centric quality, as the latter may depend on factors that go beyond the characteristics of the recommendation algorithm itself.…”

Exploring the Semantic Gap for Movie Recommendations

“…Our explanation is that the low accuracy of visual-only recommendations negatively a ects the user opinion on the other metrics. A possible interpretation of this result is to consider that previous studies con rmed a mismatch between o ine and online quality of recommendations [7,23].…”

“…is result is partially in contrast with the previous study, in which novelty and diversity with mise-en-scène recommendations are signi cantly be er than with traditional a ributes. is could be explained by previous works suggesting that o ine evaluations metrics are not always good predictors of the perceived quality of recommender systems [7,23]. However, when hybridizing recommendations based on low-level mise-en-scène features and highlevel semantic a ributes, they are perceived as be er along all metrics.…”

“…e detailed procedure used to measure di erent recommendation quality metrics is similar to the one used in previous works [7,11] maintaining compatibility with results published in other research papers.…”

Combining Review Text Content and Reviewer-Item Rating Matrix to Predict Review Rating

“…Up to now, some researches combining ratings and text reviews have been applied to recommend system [29, 30]. For example, Cremonesi et al proposed a hotel recommender algorithm (Interleave), which provides recommendations based on the text reviews and ratings [29].…”