On the recommending of citations for research papers

Abstract

Collaborative filtering has proven to be valuable for recommending items in many different domains. In this paper, we explore the use of collaborative filtering to recommend research papers, using the citation web between papers to create the ratings matrix. Specifically, we tested the ability of collaborative filtering to recommend citations that would be suitable additional references for a target research paper. We investigated six algorithms for selecting citations, evaluating them through offline experiments against a database of over 186,000 research papers contained in ResearchIndex. We also performed an online experiment with over 120 users to gauge user opinion of the effectiveness of the algorithms and of the utility of such recommendations for common research tasks. We found large differences in the accuracy of the algorithms in the offline experiment, especially when balanced for coverage. In the online experiment, users felt they received quality recommendations, and were enthusiastic about the idea of receiving recommendations in this domain.

Towards reproducibility in recommender-systems research

“…The researchers proposed and evaluated several contentbased filtering (CBF) and collaborative filtering (CF) approaches for research-paper recommendations. In one experiment, CF and CBF performed similarly well (McNee et al 2002). In other experiments, CBF outperformed CF (Dong et al 2009;McNee et al 2002;Torres et al 2004), and in some more experiments CF outperformed CBF (Ekstrand et al 2010;McNee et al 2006;Torres et al 2004).…”

“…21 % of the approaches introduced in the research papers were not evaluated. Of the evaluated approaches, 69 % were evaluated using offline evaluations, a method with serious shortcomings (Beel et al 2013c;Cremonesi et al 2011Cremonesi et al , 2012Hersh et al 2000a, b;Jannach et al 2013;Knijnenburg et al 2012;McNee et al 2002;Said 2013;Turpin and Hersh 2001). Furthermore, many approaches were evaluated against trivial (if any) baselines.…”

“…Some researchers criticize offline evaluations as being unsuitable for evaluating recommender systems because they ignore human factors and are often based on biased ground-truths (Beel et al 2013c;Cremonesi et al 2011Cremonesi et al , 2012Hersh et al 2000a, b;Jannach et al 2013;Knijnenburg et al 2012;McNee et al 2002;Said 2013;Turpin and Hersh 2001). If the critics are right, contradictions between offline experiments and user studies are to be expected because offline evaluations would provide meaningless results.…”

“…Herlocker et al (2004) wrote an article on how to evaluate collaborative-filtering approaches. Various authors showed that offline and online evaluations often provide contradictory results (Cremonesi et al 2012;McNee et al 2002), and several more papers about various aspects of recommender-system evaluation have been published (Amatriain et al 2009;Beel and Langer 2015;Bollen and Rocha 2000;Bogers and van den Bosch 2007;Cremonesi et al 2011;Domingues Garcia et al 2012;Ge et al 2010;Hayes et al 2002;Hofmann et al 2014;Jannach et al 2012;Knijnenburg et al 2011Knijnenburg et al , 2012Konstan and Riedl 2012;Manouselis and Verbert 2013;Pu et al 2011Pu et al , 2012Said 2013;Shani and Gunawardana 2011). However, while many of the findings in these papers are important with respect to reproducibility, the authors did not mention or discuss their findings in the context of reproducibility.…”

A pragmatic procedure to support the user-centric evaluation of recommender systems

“…Increasingly, industry and academic researchers agree that the ultimate goal of recommenders is to help users make better decisions, and that high accuracy in itself does not guarantee this aim [8,9,11,12]. In fact, the recommender with the highest accuracy may not even be the most satisfying [7,15]. A plethora of other factors, such as the composition of the recommendation set [1], the preference elicitation method [13], and personal considerations (e.g.…”