User perception of differences in recommender algorithms

Ekstrand, Michael D.; Harper, F. Maxwell; Willemsen, Martijn C.; Konstan, Joseph A.

doi:10.1145/2645710.2645737

Cited by 134 publications

(72 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Seen in this way, minimal necessary data is a continuation of an existing evolution. Ensuring User-centered Recommendation Recently, research studies have demonstrated that algorithm accuracy does not necessary play a dominant role in the reception of a recommender system by users [7,10]. If performance improvements achieved by using more data to train a recommender system are too slight or subtle for users to notice, the additional data is adding no value, and should not be used.…”

Section: Background and Motivationmentioning

confidence: 99%

Towards Minimal Necessary Data: The Case for Analyzing Training Data Requirements of Recommender Algorithms

Larson¹,

Zito²,

Loni³

et al.

Boise State ScholarWorks

View full text Add to dashboard Cite

This paper states the case for the principle of minimal necessary data: If two recommender algorithms achieve the same effectiveness, the better algorithm is the one that requires less user data. Applying this principle involves carrying out training data requirements analysis, which we argue should be adopted as best practice for the development and evaluation of recommender algorithms. We take the position that responsible recommendation is recommendation that serves the people whose data it uses. To minimize the imposition on users' privacy, it is important that a recommender system does not collect or store more user information than it absolutely needs. Further, algorithms using minimal necessary data reduce training time and address the cold start problem. To illustrate the trade-off between training data volume and accuracy, we carry out a set of classic recommender system experiments. We conclude that consistently applying training data requirements analysis would represent a relatively small change in researchers' current practices, but a large step towards more responsible recommender systems. ACM Reference format:

show abstract

Section: Background and Motivationmentioning

confidence: 99%

Towards Minimal Necessary Data: The Case for Analyzing Training Data Requirements of Recommender Algorithms

Larson¹,

Zito²,

Loni³

et al.

Boise State ScholarWorks

View full text Add to dashboard Cite

show abstract

“…We compute novelty by computing the mean popularity rank of the items recommended to the user. Finally, coverage is computed by calculating the percentage of pairs of < user , mo ie > for which we can predict a rating [2,15,17]. All metrics range from 0% to 100%, where 100% is the best.…”

Section: Study A: O Line Experimentsmentioning

confidence: 99%

“…Perceived quality is the degree to which the users judge recommendations positively and appreciate the overall experience with the recommender system. To be er scope our research, we operationalize this notion in terms of fours metrics: perceived accuracy, novelty, diversity, and overall user satisfaction as de ned in [15,22,27]: Perceived accuracy (also called Relevance) measures how much the recommendations match the users' interests, preferences and tastes; Diversity measures how much users perceive recommendations as di erent from each other, e.g. movies from different genres; Novelty measures the extent to which users receive new recommended movies; Overall Users' Satisfaction measures the global users' feeling of the experience with the recommender system.…”

Section: Study B: Online Evaluationmentioning

confidence: 99%

See 1 more Smart Citation

Exploring the Semantic Gap for Movie Recommendations

Elahi

Deldjoo

Moghaddam

et al. 2017

Proceedings of the Eleventh ACM Conference on Recommender Systems

View full text Add to dashboard Cite

In the last years, there has been much a ention given to the semantic gap problem in multimedia retrieval systems. Much e ort has been devoted to bridge this gap by building tools for the extraction of high-level, semantics-based features from multimedia content, as low-level features are not considered useful because they deal primarily with representing the perceived content rather than the semantics of it.In this paper, we explore a di erent point of view by leveraging the gap between low-level and high-level features. We experiment with a recent approach for movie recommendation that extract lowlevel Mise-en-Scène features from multimedia content and combine it with high-level features provided by the wisdom of the crowd.To this end, we rst performed an o ine performance assessment by implementing a pure content-based recommender system with three di erent versions of the same algorithm, respectively based on (i) conventional movie a ributes, (ii) mise-en-scène features, and (iii) a hybrid method that interleaves recommendations based on movie a ributes and mise-en-scène features. In a second study, we designed an empirical study involving 100 subjects and collected data regarding the quality perceived by the users. Results from both studies show that the introduction of mise-en-scène features in conjunction with traditional movie a ributes improves both o ine and online quality of recommendations.

show abstract

“…On-line volunteers are participants who voluntarily responded to a call for participants, via e-mail or post mail. The number of participants often depends on the credibility of the experiment conductor: if the conductor can obtain the cooperation of a real service provider, a relatively large number of users tend to become involved [21], [96]. Recently, a decent number of users can be gathered through cloud sourcing, such as that through Amazon Mechanical Turk (https://www.mturk.com).…”

Section: Small-sized User Studiesmentioning

confidence: 99%

A Survey on Recommendation Methods Beyond Accuracy

Han

Yamana

2017

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYIn recommending to another individual an item that one loves, accuracy is important, however in most cases, focusing only on accuracy generates less satisfactory recommendations. Studies have repeatedly pointed out that aspects that go beyond accuracy-such as the diversity and novelty of the recommended items-are as important as accuracy in making a satisfactory recommendation. Despite their importance, there is no global consensus about definitions and evaluations regarding beyondaccuracy aspects, as such aspects closely relate to the subjective sensibility of user satisfaction. In addition, devising algorithms for this purpose is difficult, because algorithms concurrently pursue the aspects in trade-off relation (i.e., accuracy vs. novelty). In the aforementioned situation, for researchers initiating a study in this domain, it is important to obtain a systematically integrated view of the domain. This paper reports the results of a survey of about 70 studies published over the last 15 years, each of which addresses recommendations that consider beyond-accuracy aspects. From this survey, we identify diversity, novelty, and coverage as important aspects in achieving serendipity and popularity unbiasedness-factors that are important to user satisfaction and business profits, respectively. The five major groups of algorithms that tackle the beyond-accuracy aspects are multi-objective, modified collaborative filtering (CF), clustering, graph, and hybrid; we then classify and describe algorithms as per this typology. The off-line evaluation metrics and user studies carried out by the studies are also described. Based on the survey results, we assert that there is a lot of room for research in the domain. Especially, personalization and generalization are considered important issues that should be addressed in future research (e.g., automatic per-user-trade-off among the aspects, and properly establishing beyond-accuracy aspects for various types of applications or algorithms).

show abstract

User perception of differences in recommender algorithms

Cited by 134 publications

References 21 publications

Towards Minimal Necessary Data: The Case for Analyzing Training Data Requirements of Recommender Algorithms

Towards Minimal Necessary Data: The Case for Analyzing Training Data Requirements of Recommender Algorithms

Exploring the Semantic Gap for Movie Recommendations

A Survey on Recommendation Methods Beyond Accuracy

Contact Info

Product

Resources

About