Research paper recommender system evaluation

Beel, Joeran; Langer, Stefan; Genzmehr, Marcel; Gipp, Béla; Breitinger, Corinna; Nürnberger, Andreas

doi:10.1145/2532508.2532512

Cited by 116 publications

(26 citation statements)

References 86 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We reviewed 89 research-paper recommender system approaches and found some shared shortcomings in the evaluations (Beel et al 2013b(Beel et al , 2015a. 21 % of the approaches introduced in the research papers were not evaluated.…”

Section: Impact Of Differences In the Evaluationsmentioning

confidence: 99%

“…In the recommender-systems community, we found that reproducibility is rarely given, particularly in the research-paper recommender-system community (Beel et al 2013b(Beel et al , 2015a. In a review of 89 evaluations of research-paper recommender-systems, we found several cases in which very slight variations in the experimental set-up led to surprisingly different outcomes.…”

Section: Introductionmentioning

confidence: 97%

See 1 more Smart Citation

Towards reproducibility in recommender-systems research

Beel

Breitinger

Langer

et al. 2016

User Model User-Adap Inter

Self Cite

View full text Add to dashboard Cite

Numerous recommendation approaches are in use today. However, comparing their effectiveness is a challenging task because evaluation results are rarely reproducible. In this article, we examine the challenge of reproducibility in recommender-system research. We conduct experiments using Plista's news recommender system, and Docear's research-paper recommender system. The experiments show that there are large discrepancies in the effectiveness of identical recommendation approaches in only slightly different scenarios, as well as large discrepancies for slightly different approaches in identical scenarios. For example, in one newsrecommendation scenario, the performance of a content-based filtering approach User-Adapted Interaction : umuai ; 26 (2016), 1. -S. 69-101 https://dx.doi.org/10.1007 was twice as high as the second-best approach, while in another scenario the same content-based filtering approach was the worst performing approach. We found several determinants that may contribute to the large discrepancies observed in recommendation effectiveness. Determinants we examined include user characteristics (gender and age), datasets, weighting schemes, the time at which recommendations were shown, and user-model size. Some of the determinants have interdependencies. For instance, the optimal size of an algorithms' user model depended on users' age. Since minor variations in approaches and scenarios can lead to significant changes in a recommendation approach's performance, ensuring reproducibility of experimental results is difficult. We discuss these findings and conclude that to ensure reproducibility, the recommender-system community needs to (1) survey other research fields and learn from them, (2) find a common understanding of reproducibility, (3) identify and understand the determinants that affect reproducibility, (4) conduct more comprehensive experiments, (5) modernize publication practices, (6) foster the development and use of recommendation frameworks, and (7) establish best-practice guidelines for recommender-systems research.

show abstract

Section: Impact Of Differences In the Evaluationsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 97%

Towards reproducibility in recommender-systems research

Beel

Breitinger

Langer

et al. 2016

User Model User-Adap Inter

Self Cite

View full text Add to dashboard Cite

show abstract

“…These recommended products are recognized on the basis of the usage and purchasing patterns of people in the past. In the field of scientific literature search, the exponential increase in the number of published articles necessitates having a recommender system to help scientists explore relevant and recent papers quickly [ 6–8 ]. Recommender systems are important in the world of commercial applications too.…”

Section: Introductionmentioning

confidence: 99%

Tool recommender system in Galaxy using deep learning

et al. 2021

View full text Add to dashboard Cite

Background Galaxy is a web-based and open-source scientific data-processing platform. Researchers compose pipelines in Galaxy to analyse scientific data. These pipelines, also known as workflows, can be complex and difficult to create from thousands of tools, especially for researchers new to Galaxy. To help researchers with creating workflows, a system is developed to recommend tools that can facilitate further data analysis. Findings A model is developed to recommend tools using a deep learning approach by analysing workflows composed by researchers on the European Galaxy server. The higher-order dependencies in workflows, represented as directed acyclic graphs, are learned by training a gated recurrent units neural network, a variant of a recurrent neural network. In the neural network training, the weights of tools used are derived from their usage frequencies over time and the sequences of tools are uniformly sampled from training data. Hyperparameters of the neural network are optimized using Bayesian optimization. Mean accuracy of 98% in recommending tools is achieved for the top-1 metric. Conclusions The model is accessed by a Galaxy API to provide researchers with recommended tools in an interactive manner using multiple user interface integrations on the European Galaxy server. High-quality and highly used tools are shown at the top of the recommendations. The scripts and data to create the recommendation system are available under MIT license at https://github.com/anuprulez/galaxy_tool_recommendation.

show abstract

“…These methods mainly include content-based methods [5], collaborative filtering methods [6], and hybrid methods [7]. These methods come with legacy problems of data sparsity and cold start [8,9], which are under active research focused mainly on optimizing recommendation efficiency and accuracy based on user interests and preferences [10,11]. However, to deliver mobile marketing contents to specific users accurately by recommendation systems, there are still huge challenges: different from traditional online marketing, mobile marketing introduces stronger interactivity on smaller user interfaces [12,13].…”

Section: Introductionmentioning

confidence: 99%

Research on Mobile Marketing Recommendation Method Incorporating Layout Aesthetic Preference for Sustainable m-Commerce

2020

View full text Add to dashboard Cite

The strong interactivity and size limitation of the mobile interface calls for the utilization of users’ aesthetic preferences to provide better mobile marketing recommendations in order to promote the sustainable development of m-commerce. Existing studies mostly focus on matching user interests by analyzing marketing content properties. The studies for utilizing the layout information and user aesthetic preferences for the layout of the mobile marketing interface from an aesthetic perspective are insufficient. This paper proposes a mobile marketing recommendation method (LAPR) that incorporates layout aesthetic preferences. Based on the traditional content-based and collaborative filtering recommendation methods, this method introduces users’ aesthetic preferences for interface layout into a mobile marketing recommender. From an aesthetic perspective, a new interface layout design quantification method, a user aesthetic preference similarity measurement model, and a recommendation result ranking method are designed. Experiments show that compared to traditional methods, LAPR is significantly higher in recommendation precision in the task for recommending the same content and outperforms traditional methods in recall, precision, and F-metrics in the common recommendation task. We conclude that incorporating aesthetic preference for layout can improve mobile marketing recommendation quality and promote the sustainable development of m-commerce.

show abstract

Research paper recommender system evaluation

Cited by 116 publications

References 86 publications

Towards reproducibility in recommender-systems research

Towards reproducibility in recommender-systems research

Tool recommender system in Galaxy using deep learning

Research on Mobile Marketing Recommendation Method Incorporating Layout Aesthetic Preference for Sustainable m-Commerce

Contact Info

Product

Resources

About