A Revisiting Study of Appropriate Offline Evaluation for Top- N Recommendation Algorithms

Abstract

In recommender system, top- N recommendation is an important task with implicit feedback data. Although the recent success of deep learning largely pushes forward the research on top- N recommendation, there are increasing concerns on appropriate evaluation of recommendation algorithms. It becomes emergent to study how recommendation algorithms can be reliably evaluated and thoroughly verified. This work presents a large-scale, systematic study on six important factors from three aspects for evaluating recommender systems. We carefully select twelve top- N recommendation algorithms and eight recommendation datasets. Our experiments are carefully designed and extensively conducted with these algorithms and datasets. In particular, all the experiments in our work are implemented based on an open-sourced recommendation library, Recbole [139], which ensures the reproducibility and reliability of our results. Based on the large-scale experiments and detailed analysis, we derive several key findings on the experimental settings for evaluating recommender systems. Our findings show that some settings can lead to substantial or significant differences in performance ranking of the compared algorithms. In response to recent evaluation concerns, we also provide several suggested settings that are specially important for performance comparison.

Time-Aware Explainable Recommendation via Updating Enabled Online Prediction

“…Therefore, the recommender results and explanations of our proposed online prediction method will be more convincing. The impact of data leakage is recognized by existing works [ 14 , 15 ], but we are the first to offer a comprehensive critical study on this issue under the explainable recommendation scenario.…”

“…The propose of data splitting is to divide the original dataset into training and test sets. As their significant impact of recommendation performance, different data splitting strategies of recommendation were widely evaluated [ 14 , 28 ]. And according to the choices of data ordering and splitting, there are mainly four types of them, including random ordering ratio-based splitting, random ordering leave-one-out splitting, temporal ordering ratio-based splitting, and temporal ordering leave-one-out splitting.…”

“…However, most existing KG-based recommendation techniques use static knowledge graph with offline prediction in which models are trained once while used forever [ 13 , 14 ]. There are two main issues of these methods, namely the data leakage issue and the model aging issue.…”

Impact of temporal context on recommender systems along global timeline

“…In offline evaluation, the pre-collected user-item interactions are split into training and test sets. The splitting strategies can be categorized as random-split, leaveone-out split and temporal split [15,17,21,[143][144][145]. Random-split is to randomly Other than the general evaluation setting mentioned, prequential evaluation is used to evaluate recommender system in a streaming environment [112,113,146].…”

“…To achieve this objective, we analyse the data splitting strategies in offline evaluation, which have garnered considerable interests from other researchers [15][16][17][18][19][20][21][22].…”