Towards reproducibility in recommender-systems research

Abstract

Numerous recommendation approaches are in use today. However, comparing their effectiveness is a challenging task because evaluation results are rarely reproducible. In this article, we examine the challenge of reproducibility in recommender-system research. We conduct experiments using Plista's news recommender system, and Docear's research-paper recommender system. The experiments show that there are large discrepancies in the effectiveness of identical recommendation approaches in only slightly different scenarios, as well as large discrepancies for slightly different approaches in identical scenarios. For example, in one newsrecommendation scenario, the performance of a content-based filtering approach User-Adapted Interaction : umuai ; 26 (2016), 1. -S. 69-101 https://dx.doi.org/10.1007 was twice as high as the second-best approach, while in another scenario the same content-based filtering approach was the worst performing approach. We found several determinants that may contribute to the large discrepancies observed in recommendation effectiveness. Determinants we examined include user characteristics (gender and age), datasets, weighting schemes, the time at which recommendations were shown, and user-model size. Some of the determinants have interdependencies. For instance, the optimal size of an algorithms' user model depended on users' age. Since minor variations in approaches and scenarios can lead to significant changes in a recommendation approach's performance, ensuring reproducibility of experimental results is difficult. We discuss these findings and conclude that to ensure reproducibility, the recommender-system community needs to (1) survey other research fields and learn from them, (2) find a common understanding of reproducibility, (3) identify and understand the determinants that affect reproducibility, (4) conduct more comprehensive experiments, (5) modernize publication practices, (6) foster the development and use of recommendation frameworks, and (7) establish best-practice guidelines for recommender-systems research.

Are we really making much progress? A worrying analysis of recent neural recommendation approaches

“…• A working version of the source code is available or the code only has to be modified in minimal ways to work correctly. 3 • At least one dataset used in the original paper is available. A further requirement here is that either the originally-used train-test splits are publicly available or that they can be reconstructed based on the information in the paper.…”

“…Precisely speaking, we used a mix of replication and reproduction[12,35], i.e., we used both artifacts provided by the authors and our own artifacts. For the sake of readability, we will only use the term "reproducibility" in this paper 3. We did not apply modifications to the core algorithms.…”

Online Evaluations for Everyone: Mr. DLib’s Living Lab for Scholarly Recommendations

“…In the long-run, we hope to provide a platform to the information retrieval, digital library, and recommender systems community that helps conducting more reproducible and robust research in real-world scenarios [34,35]. To achieve this, we plan to add more partners on both sidesplatform partners who provide access to real users, and research partners who evaluate their novel algorithms via the living lab.…”

A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms

“…Unfortunately, university-based researchers struggle unless they closely collaborate with industry (e.g., [7]) or develop their own infrastructure and user base (e.g., [1]). Without online testing opportunities open to the research communities, they cannot employ online evaluation on a larger scale, which is the de-facto standard evaluation methodology in industry.…”