A pragmatic procedure to support the user-centric evaluation of recommender systems

Knijnenburg, Bart P.; Willemsen, Martijn C.; Kobsa, Alfred

doi:10.1145/2043932.2043993

Cited by 55 publications

(33 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In fact, users provide rather high average rating scores to all the algorithms (KNN: 4.05; graph-based: 4.1; MF: 3.95; all out of 5). Since our sample size was sufficiently large at 20 users per algorithm [5], we suggest this to show that the users were satisfied with the accuracy of the recommendations, regardless of the type of algorithm that generated them. Nevertheless, the user-centric evaluation results ( Figure 5) show that the graph-based recommender received a somewhat greater average rating score for perceived usefulness, novelty and serendipity of the recommendations by users, compared to the other two algorithms.…”

Section: Discussionmentioning

confidence: 99%

“…The questionnaire was designed to reflect how actual users perceive and appreciate the recommendations they receive, taking into account important aspects in user perception when running recommender systems' user studies [5], [28]. We asked the participants to answer six short questions by expressing their level of agreement with each of the questions.…”

Section: Questionnairementioning

confidence: 99%

“…We managed to obtain the same number of participants for the three algorithms, twenty for each. For user-centric evaluations in the recommender systems area, it has been claimed that "at least twenty (20) users" per condition is adequate to make a user-centric evaluation statistically sound [5].…”

Section: Questionnairementioning

confidence: 99%

“…These authors also emphasized that researchers should "study recommenders from a user-centric perspective to make them not only accurate and helpful, but also a pleasure to use" [4]. The second change has been introduced as "a broadening of the scope of research regarding the system aspects to investigate beyond just the algorithm of the recommender" [3], [5]. Following this, McNee et al suggest researchers to also study the aspects of "Human-Recommender Interaction" [6].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

User-Centric Evaluation of Recommender Systems in Social Learning Platforms: Accuracy is Just the Tip of the Iceberg

Fazeli

Drachsler

Bitter-Rijpkema

et al. 2018

IEEE Trans. Learning Technol.

View full text Add to dashboard Cite

Abstract-Recommender systems provide users with content they might be interested in. Conventionally, recommender systems are evaluated mostly by using prediction accuracy metrics only. But the ultimate goal of a recommender system is to increase user satisfaction. Therefore, evaluations that measure user satisfaction should be also performed before deploying a recommender system to a real target environment. Such evaluations are laborious and complicated compared to the traditional, data-centric evaluations, though. In this study, we investigate the added value of user-centric evaluations and how user satisfaction of a recommender system is related to its performance in terms of accuracy metrics. We conduct both a data-centric evaluation and a user-centric evaluation on the same data collected from an authentic social learning platform. Our findings suggest that user-centric evaluation results are not necessarily in line with data-centric evaluation results. We conclude that the traditional evaluation of recommender systems in terms of prediction accuracy does not suffice to judge performance of recommender systems on the user side. Moreover, the user-centric evaluation provides valuable insights on how candidate algorithms perform on each of the five quality metrics: usefulness, accuracy, novelty, diversity, and serendipity of the recommendations.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Questionnairementioning

confidence: 99%

Section: Questionnairementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

User-Centric Evaluation of Recommender Systems in Social Learning Platforms: Accuracy is Just the Tip of the Iceberg

Fazeli

Drachsler

Bitter-Rijpkema

et al. 2018

IEEE Trans. Learning Technol.

View full text Add to dashboard Cite

show abstract

“…Various authors showed that offline and online evaluations often provide contradictory results [93,269,280]. Many papers about various aspects of recommender-system evaluation have been published [226,268,275,280,283,[298][299][300]. However, while many of the findings in these papers are important with respect to reproducibility, none of the authors mentioned or discussed their findings in the context of reproducibility.…”

Section: Reproducibility and The Butterfly Effectmentioning

confidence: 99%

Research-paper recommender systems: a literature survey

et al. 2015

View full text Add to dashboard Cite

In the last 16 years, more than 200 research articles were published about research-paper recommender systems. We reviewed these articles and present some descriptive statistics in this paper, as well as a discussion about the major advancements and shortcomings and an overview of the most common recommendation concepts and approaches. We found that more than half of the recommendation approaches applied content-based filtering (55 %). Collaborative filtering was applied by only 18 % of the reviewed approaches, and graph-based recommendations by 16 %. Other recommendation concepts included stereotyping, item-centric recommendations, and hybrid recommendations. The content-based filtering approaches mainly utilized papers that the users had authored, tagged, browsed, or downloaded. TF-IDF was the most frequently applied weighting scheme. In addition to simple terms, n-grams, topics, and citations were utilized to model users' information needs. Our review revealed some shortcomings of the current research. First, it remains unclear which recommendation concepts and approaches are the most promising. For instance, researchers reported different results on the performance of contentbased and collaborative filtering. Sometimes content-based filtering performed better than collaborative filtering and Linnaeus University, Kalmar, Sweden sometimes it performed worse. We identified three potential reasons for the ambiguity of the results. (A) Several evaluations had limitations. They were based on strongly pruned datasets, few participants in user studies, or did not use appropriate baselines. (B) Some authors provided little information about their algorithms, which makes it difficult to re-implement the approaches. Consequently, researchers use different implementations of the same recommendations approaches, which might lead to variations in the results. (C) We speculated that minor variations in datasets, algorithms, or user populations inevitably lead to strong variations in the performance of the approaches. Hence, finding the most promising approaches is a challenge. As a second limitation, we noted that many authors neglected to take into account factors other than accuracy, for example overall user satisfaction. In addition, most approaches (81 %) neglected the user-modeling process and did not infer information automatically but let users provide keywords, text snippets, or a single paper as input. Information on runtime was provided for 10 % of the approaches. Finally, few research papers had an impact on research-paper recommender systems in practice. We also identified a lack of authority and long-term research interest in the field: 73 % of the authors published no more than one paper on research-paper recommender systems, and there was little cooperation among different co-author groups. We concluded that several actions could improve the research landscape: developing a common evaluation framework, agreement on the information to include in research papers, a stronger focus on non-accuracy aspect...

show abstract

Evaluating Recommender Systems with User Experiments

Knijnenburg

Willemsen

2015

Recommender Systems Handbook

View full text Add to dashboard Cite

A pragmatic procedure to support the user-centric evaluation of recommender systems

Cited by 55 publications

References 17 publications

User-Centric Evaluation of Recommender Systems in Social Learning Platforms: Accuracy is Just the Tip of the Iceberg

User-Centric Evaluation of Recommender Systems in Social Learning Platforms: Accuracy is Just the Tip of the Iceberg

Research-paper recommender systems: a literature survey

Evaluating Recommender Systems with User Experiments

Contact Info

Product

Resources

About