Overview of NewsREEL’16: Multi-dimensional Evaluation of Real-Time Stream-Recommendation Algorithms

Abstract

Abstract. Successful news recommendation requires facing the challenges of dynamic item sets, contextual item relevance, and of fulfilling non-functional requirements, such as response time. The CLEF NewsREEL challenge is a campaignstyle evaluation lab allowing participants to tackle news recommendation and to optimize and evaluate their recommender algorithms both online and offline. In this paper, we summarize the objectives and challenges of NewsREEL 2016. We cover two contrasting perspectives on the challenge: that of the operator (the business providing recommendations) and that of the challenge participant (the researchers developing recommender algorithms). In the intersection of these perspectives, new insights can be gained on how to effectively evaluate real-time stream recommendation algorithms.

CLEF 2017 NewsREEL Overview: A Stream-Based Recommender Task for Evaluation and Education

“…In 2013, plista released a data set specifically for news recommender systems [14]. Subsequently, multiple updates of the data set have been released in scope of CLEF NewsREEL [13,18,15]. Li et al [19] model news recommendation as contextual bandit problem.…”

Online Evaluations for Everyone: Mr. DLib’s Living Lab for Scholarly Recommendations

“…Living labs are available in information retrieval and for many recommender-system domains, particularly news [4][5][6], and they attracted dedicated workshops [7]. There is also work on living labs in the context of search and browsing behavior in digital libraries [8].…”