Investigating the Persuasion Potential of Recommender Systems from a Quality Perspective

Abstract

Recommender Systems (RSs) help users search large amounts of digital contents and services by allowing them to identify the items that are likely to be more attractive or useful. RSs play an important persuasion role, as they can potentially augment the users' trust towards in an application and orient their decisions or actions towards specific directions. This article explores the persuasiveness of RSs, presenting two vast empirical studies that address a number of research questions.First, we investigate if a design property of RSs, defined by the statistically measured quality of algorithms, is a reliable predictor of their potential for persuasion. This factor is measured in terms of perceived quality, defined by the overall satisfaction, as well as by how users judge the accuracy and novelty of recommendations. For our purposes, we designed an empirical study involving 210 subjects and implemented seven full-sized versions of a commercial RS, each one using the same interface and dataset (a subset of Netflix), but each with a different recommender algorithm. In each experimental configuration we computed the statistical quality (recall and F-measures) and collected data regarding the quality perceived by 30 users. The results show us that algorithmic attributes are less crucial than we might expect in determining the user's perception of an RS's quality, and suggest that the user's judgment and attitude towards a recommender are likely to be more affected by factors related to the user experience.Second, we explore the persuasiveness of RSs in the context of large interactive TV services. We report a study aimed at assessing whether measurable persuasion effects (e.g., changes of shopping behavior) can be achieved through the introduction of a recommender. Our data, collected for more than one year, allow us to conclude that, (1) the adoption of an RS can affect both the lift factor and the conversion rate, determining an increased volume of sales and influencing the user's decision to actually buy one of the recommended products, (2) the introduction of an RS tends to diversify purchases and orient users towards less obvious choices (the long tail), and (3) the perceived novelty of recommendations is likely to be more influential than their perceived accuracy.Overall, the results of these studies improve our understanding of the persuasion phenomena induced by RSs, and have implications that can be of interest to academic scholars, designers, and adopters of this class of systems.

User-Centric vs. System-Centric Evaluation of Recommender Systems

“…We can claim that, in the e-tourism domain, system-centric quality attributes are good predictors of how the users perceive the quality of a recommender algorithm. This result is apparently in contrast with previous works [6,19,22,27] which state that user satisfaction is not correlated with accuracy of algorithms.…”

“…The comparison of the evaluation outcomes shows that system-centric and usercentric metrics lead to consistent results, in contrast with past studies in e-commerce [5,6], and suggests that in the online hotel booking domain system-centric accuracy measures are good predictors of the beneficial effect of personalized recommendations on user's decision making. Our findings pinpoints that the relationship between the system-centric and user-centric metrics may depend on the business sector, is more complex that we may expect, and is a challenging issues that deserves further research.…”

“…The same ranking strategy is adopted by most online hotel booking systems such as TripAdvisor, Expedia, and Venere. ─ PureSVD is a collaborative algorithm based on matrix-factorization; previous research shows that its accuracy is one of the best in the movie domain [6]. ─ DirectContent recommends hotels whose content is similar to the content of hotels the user has rated [18].…”

“…The methodology adopted to measure the two variables is the same used in many works on recommender system -e.g., [6,16]. Ratings in the dataset were randomly split into two subsets: training set (90% of the ratings) and test set (10% of the ratings).…”

“…On the contrary, system-centric evaluation has the advantage to be immediate, economical and easy to perform on several domains and with multiple algorithms. Recently, many researchers have argued that the system-centric evaluation of RSs in e-commerce applications does not always correlate with how the users perceive the value of recommendations [2,5,6,19,22,27]. This may happen because system-centric evaluation cannot reliably measure non-accuracy metrics such as novelty -the extension to which recommendations are perceived as new -which more reflects the user and business dimensions.…”