Benchmarking News Recommendations in a Living Lab

Self Cite

Abstract. Successful news recommendation requires facing the challenges of dynamic item sets, contextual item relevance, and of fulfilling non-functional requirements, such as response time. The CLEF NewsREEL challenge is a campaignstyle evaluation lab allowing participants to tackle news recommendation and to optimize and evaluate their recommender algorithms both online and offline. In this paper, we summarize the objectives and challenges of NewsREEL 2016. We cover two contrasting perspectives on the challenge: that of the operator (the business providing recommendations) and that of the challenge participant (the researchers developing recommender algorithms). In the intersection of these perspectives, new insights can be gained on how to effectively evaluate real-time stream recommendation algorithms.

Section: Problem Descriptionmentioning

confidence: 99%

“…We address this issue by using the evaluation framework Idomaar 12 . The framework is a recommender system reference framework developed in the settings of the European Project CrowdRec 13 . It builds reproducible computing environments based on virtual machines having an exactly defined software environment based on PUPPET.…”

Section: Ctr-related Metricsmentioning

confidence: 99%

Overview of NewsREEL’16: Multi-dimensional Evaluation of Real-Time Stream-Recommendation Algorithms

Kille

Lommatzsch

Gebremeskel

et al. 2016

Self Cite

“…In 2015, we participated in the Living Lab setting of the CLEF News Recommendations Evaluation Lab (NewsREEL) [4]. CLEF NewsREEL is a campaign-like online recommender system evaluation, where participants in need of testing their algorithms are connected with real-life online information portals in need of recommendation services.…”

Section: Approachmentioning

confidence: 99%

Random Performance Differences Between Online Recommender System Algorithms

Gebremeskel

Vries²

2016

Abstract. In the evaluation of recommender systems, the quality of recommendations made by a newly proposed algorithm is compared to the state-of-the-art, using a given quality measure and dataset. Validity of the evaluation depends on the assumption that the evaluation does not exhibit artefacts resulting from the process of collecting the dataset. The main difference between online and offline evaluation is that in the online setting, the user's response to a recommendation is only observed once. We used the NewsREEL challenge to gain a deeper understanding of the implications of this difference for making comparisons between different recommender systems. The experiments aim to quantify the expected degree of variation in performance that cannot be attributed to differences between systems. We classify and discuss the non-algorithmic causes of performance differences observed.

“…In the remainder of this section, we briefly outline the new recommendation use case. For a more detailed description of the recommendation scenario, the provided content and its users, the reader is referred to [9]. An overview of the approaches of last year's participants of NEWSREEL 2014 is provided in [10].…”

Section: News Recommendation Use Casementioning

confidence: 99%

Join the Living Lab: Evaluating News Recommendations in Real-Time

Hopfgartner

Brodt²

2015

Self Cite

Abstract. Participants of this tutorial learnt how to participate in CLEF NEWSREEL, a living lab for the evaluation of news recommender algorithms. Various research challenges can be addressed within NEWS-REEL, such as the development and evaluation of collaborative filtering or content-based filtering strategies. Satisfying information needs by techniques including preference elicitation, pattern recognition, and prediction, recommender systems connect the research areas information retrieval and machine learning.