Automatically building research reading lists

Ekstrand, Michael D.; Kannan, Praveen; Stemper, James A.; Butler, John T.; Konstan, Joseph A.; Riedl, John

doi:10.1145/1864708.1864740

Cited by 70 publications

(87 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this evaluation, CBF outperformed CF, contradicting the previous offline results from Torres et al In 2010, Ekstrand et al found that CBF performed worse than CF in both an offline evaluation and a user study, which again did not align with the previous findings [26].…”

Section: Reproducibility and The Butterfly Effectcontrasting

confidence: 94%

“…Some of the approaches were evaluated using both an offline evaluation and a user study. In two evaluations, results from the offline evaluations were indeed similar to results of the user studies [26,84]. However, the user studies had five and 19 participants, respectively, which led to statistically insignificant results.…”

Section: Offline Evaluationsmentioning

confidence: 51%

“…This means that offline evaluations could not reliably predict the effectiveness in the real-world use case. Interestingly, the three studies with the most participants were all conducted by the authors of TechLens [26,93,117], who are also the only authors in the field of research-paper recommender systems who discuss the potential shortcomings of offline evaluations [87]. It seems that other researchers in this field are not aware of-or chose not to address-problems associated with offline evaluations, although there has been quite a discussion outside the research-paper recommender-system community [93,269,270,[278][279][280][281].…”

Section: Offline Evaluationsmentioning

confidence: 99%

“…The approaches extracted words from the title [79,85,112], abstract [26,51,61], header [43], introduction [57], foreword [57], author-provided keywords [26,57,58], and bibliography [27], as well as from the papers' body text [65,101,112]. The approaches further extracted words from external sources, such as social tags [58,62], ACM classification tree and DMOZ categories [91,94], and citation context [51,55,65].…”

Section: Content-based Filteringmentioning

confidence: 99%

“…All variations that included Katz were about twice as good as those variations without. Bethard and Jurafsky report that a simple citation count was the most 26 Relatedness between genes was retrieved from an external data source that maintained information about gene relatedness. 27 Attribute similarity was calculated, e.g., based on the number of pages.…”

Section: Global Relevancementioning

confidence: 99%

See 4 more Smart Citations

Research-paper recommender systems: a literature survey

et al. 2015

View full text Add to dashboard Cite

In the last 16 years, more than 200 research articles were published about research-paper recommender systems. We reviewed these articles and present some descriptive statistics in this paper, as well as a discussion about the major advancements and shortcomings and an overview of the most common recommendation concepts and approaches. We found that more than half of the recommendation approaches applied content-based filtering (55 %). Collaborative filtering was applied by only 18 % of the reviewed approaches, and graph-based recommendations by 16 %. Other recommendation concepts included stereotyping, item-centric recommendations, and hybrid recommendations. The content-based filtering approaches mainly utilized papers that the users had authored, tagged, browsed, or downloaded. TF-IDF was the most frequently applied weighting scheme. In addition to simple terms, n-grams, topics, and citations were utilized to model users' information needs. Our review revealed some shortcomings of the current research. First, it remains unclear which recommendation concepts and approaches are the most promising. For instance, researchers reported different results on the performance of contentbased and collaborative filtering. Sometimes content-based filtering performed better than collaborative filtering and Linnaeus University, Kalmar, Sweden sometimes it performed worse. We identified three potential reasons for the ambiguity of the results. (A) Several evaluations had limitations. They were based on strongly pruned datasets, few participants in user studies, or did not use appropriate baselines. (B) Some authors provided little information about their algorithms, which makes it difficult to re-implement the approaches. Consequently, researchers use different implementations of the same recommendations approaches, which might lead to variations in the results. (C) We speculated that minor variations in datasets, algorithms, or user populations inevitably lead to strong variations in the performance of the approaches. Hence, finding the most promising approaches is a challenge. As a second limitation, we noted that many authors neglected to take into account factors other than accuracy, for example overall user satisfaction. In addition, most approaches (81 %) neglected the user-modeling process and did not infer information automatically but let users provide keywords, text snippets, or a single paper as input. Information on runtime was provided for 10 % of the approaches. Finally, few research papers had an impact on research-paper recommender systems in practice. We also identified a lack of authority and long-term research interest in the field: 73 % of the authors published no more than one paper on research-paper recommender systems, and there was little cooperation among different co-author groups. We concluded that several actions could improve the research landscape: developing a common evaluation framework, agreement on the information to include in research papers, a stronger focus on non-accuracy aspect...

show abstract

Section: Reproducibility and The Butterfly Effectcontrasting

confidence: 94%

Section: Offline Evaluationsmentioning

confidence: 51%