A Comparison of Offline Evaluations, Online Evaluations, and User Studies in the Context of Research-Paper Recommender Systems

Beel, Joeran; Langer, Stefan

doi:10.1007/978-3-319-24592-8_12

Cited by 71 publications

(49 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In one experiment, removing stop words increased recommendation effectiveness by 50 % (CTR = 4.16 % vs. CTR = 6.54 %) (Beel et al 2013e). In another experiment, effectiveness was almost the same (CTR = 5.94 % vs. CTR = 6.31 %) (Beel and Langer 2015). Similarly, in one experiment, the 'stereotype' recommendation approach was around 60 % more effective than in another experiment (CTR = 3.08 % vs. CTR = 4.99 %) (Beel et al 2014a(Beel et al , 2015b.…”

Section: Introductionmentioning

confidence: 72%

“…Herlocker et al (2004) wrote an article on how to evaluate collaborative-filtering approaches. Various authors showed that offline and online evaluations often provide contradictory results (Cremonesi et al 2012;McNee et al 2002), and several more papers about various aspects of recommender-system evaluation have been published (Amatriain et al 2009;Beel and Langer 2015;Bollen and Rocha 2000;Bogers and van den Bosch 2007;Cremonesi et al 2011;Domingues Garcia et al 2012;Ge et al 2010;Hayes et al 2002;Hofmann et al 2014;Jannach et al 2012;Knijnenburg et al 2011Knijnenburg et al , 2012Konstan and Riedl 2012;Manouselis and Verbert 2013;Pu et al 2011Pu et al , 2012Said 2013;Shani and Gunawardana 2011). However, while many of the findings in these papers are important with respect to reproducibility, the authors did not mention or discuss their findings in the context of reproducibility.…”

Section: Introductionmentioning

confidence: 99%

“…Another important question is "How do differences in recommendation evaluations affect recommendation effectiveness?" Although there is no agreement on the answer, there has at least been a lively discussion about it (Beel and Langer 2015;Ge et al 2010;Knijnenburg et al 2011Knijnenburg et al , 2012Konstan and Riedl 2012;Pu et al 2011Pu et al , 2012Said 2013). Therefore, we concentrate on examining differences in scenarios and approaches, which have not received much attention in the recommender-system community.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Towards reproducibility in recommender-systems research

Beel

Breitinger

Langer

et al. 2016

User Model User-Adap Inter

Self Cite

View full text Add to dashboard Cite

Numerous recommendation approaches are in use today. However, comparing their effectiveness is a challenging task because evaluation results are rarely reproducible. In this article, we examine the challenge of reproducibility in recommender-system research. We conduct experiments using Plista's news recommender system, and Docear's research-paper recommender system. The experiments show that there are large discrepancies in the effectiveness of identical recommendation approaches in only slightly different scenarios, as well as large discrepancies for slightly different approaches in identical scenarios. For example, in one newsrecommendation scenario, the performance of a content-based filtering approach User-Adapted Interaction : umuai ; 26 (2016), 1. -S. 69-101 https://dx.doi.org/10.1007 was twice as high as the second-best approach, while in another scenario the same content-based filtering approach was the worst performing approach. We found several determinants that may contribute to the large discrepancies observed in recommendation effectiveness. Determinants we examined include user characteristics (gender and age), datasets, weighting schemes, the time at which recommendations were shown, and user-model size. Some of the determinants have interdependencies. For instance, the optimal size of an algorithms' user model depended on users' age. Since minor variations in approaches and scenarios can lead to significant changes in a recommendation approach's performance, ensuring reproducibility of experimental results is difficult. We discuss these findings and conclude that to ensure reproducibility, the recommender-system community needs to (1) survey other research fields and learn from them, (2) find a common understanding of reproducibility, (3) identify and understand the determinants that affect reproducibility, (4) conduct more comprehensive experiments, (5) modernize publication practices, (6) foster the development and use of recommendation frameworks, and (7) establish best-practice guidelines for recommender-systems research.

show abstract

Section: Introductionmentioning

confidence: 72%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards reproducibility in recommender-systems research

Beel

Breitinger

Langer

et al. 2016

User Model User-Adap Inter

Self Cite

View full text Add to dashboard Cite

show abstract

“…offline evaluations and user studies. We do not discuss this issue here but refer to a recent publication, in which we showed that online evaluations are preferable over offline evaluations, and that CTR seems to be the most sensible metric for our purpose [14]. In that publication we also explain why showing only the title of the recommended papers is sufficient for our evaluation, instead of showing further information such as author name and publication year.…”

Section: Methodsmentioning

confidence: 98%

Exploring the Potential of User Modeling Based on Mind Maps

Beel¹,

Langer²,

Kapitsaki

et al. 2015

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. Mind maps have not received much attention in the user modeling and recommender system community, although mind maps contain rich information that could be valuable for user-modeling and recommender systems. In this paper, we explored the effectiveness of standard user-modeling approaches applied to mind maps. Additionally, we develop novel user modeling approaches that consider the unique characteristics of mind maps. The approaches are applied and evaluated using our mind mapping and referencemanagement software Docear. Docear displayed 430,893 research paper recommendations, based on 4,700 user mind maps, from March 2013 to August 2014. The evaluation shows that standard user modeling approaches are reasonably effective when applied to mind maps, with click-through rates (CTR) between 1.16% and 3.92%. However, when adjusting user modeling to the unique characteristics of mind maps, a higher CTR of 7.20% could be achieved. A user study confirmed the high effectiveness of the mind map specific approach with an average rating of 3.23 (out of 5), compared to a rating of 2.53 for the best baseline. Our research shows that mind map-specific user modeling has a high potential, and we hope that our results initiate a discussion that encourages researchers to do research in this field and developers to integrate recommender systems to their mind mapping tools.Keywords: mind map, user modeling, recommender systems IntroductionMind mapping is a visual technique to record and organize information, and to develop new ideas. As such, mind-maps are used for tasks like brainstorming, knowledge management, note taking, presenting, project planning, decision-making, and career planning. Originally, mind mapping was performed using pen and paper, but since the 1980s, more than one hundred software tools for aiding users in creating mind maps have evolved. These tools are used by an estimated two million users who create around five millions mind-maps every year [1]. Mindmaps received attention in various research fields. They have been used to implement a lambda calculator [2], to filter search results from Google [3], to present software requirements [4], to research how knowledgeable business school students are, and there are numerous studies about the effectiveness of mind-maps as a learning tool [5]. In the field of document engineering and text mining, mind maps were created automatically from texts [6], and have been utilized to model XML files [7]. In the field of user-modeling and recommender-systems research, mind maps have thus far received no attention. However, mind-maps typically contain structured information that reflects the interests, knowledge, and information needs of their users. In this regard, the content of mind-maps is comparable to the content of emails, web pages, and research articles [1]. Hence, we believe that mind maps should be equally well suited for user modeling as are other documents, such as emails [8], web-pages [9], and research articles [10].

show abstract

“…In one experiment to assess the effectiveness of a recommendation approach, removing stopwords increased recommendation effectiveness by 50 % [6]. In another experiment, effectiveness was almost the same [5]. Similarly, Lu et al [14] found that sometimes terms from an article's abstract performed better than terms from the article's body, but in other cases they observed the opposite.…”

mentioning

confidence: 93%

Stability Evaluation of Event Detection Techniques for Twitter

Weiler

Beel

Gipp

et al. 2016

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. Twitter continues to gain popularity as a source of up-todate news and information. As a result, numerous event detection techniques have been proposed to cope with the steadily increasing rate and volume of social media data streams. Although most of these works conduct some evaluation of the proposed technique, comparing their effectiveness is a challenging task. In this paper, we examine the challenges to reproducing evaluation results for event detection techniques. We apply several event detection techniques and vary four parameters, namely time window (15 vs. 30 vs. 60 mins), stopwords (include vs. exclude), retweets (include vs. exclude), and the number of terms that define an event (1...5 terms). Our experiments use real-world Twitter streaming data and show that varying these parameters alone significantly influences the outcomes of the event detection techniques, sometimes in unforeseen ways. We conclude that even minor variations in event detection techniques may lead to major difficulties in reproducing experiments.

show abstract

A Comparison of Offline Evaluations, Online Evaluations, and User Studies in the Context of Research-Paper Recommender Systems

Cited by 71 publications

References 26 publications

Towards reproducibility in recommender-systems research

Towards reproducibility in recommender-systems research

Exploring the Potential of User Modeling Based on Mind Maps

Stability Evaluation of Event Detection Techniques for Twitter

Contact Info

Product

Resources

About