A Critical Study on Data Leakage in Recommender System Offline Evaluation

Abstract

Recommender models are hard to evaluate, particularly under offline setting. In this paper, we provide a comprehensive and critical analysis of the data leakage issue in recommender system offline evaluation. Data leakage is caused by not observing global timeline in evaluating recommenders e.g., train/test data split does not follow global timeline. As a result, a model learns from the user-item interactions that are not expected to be available at prediction time. We first show the temporal dynamics of user-item interactions along global timeline, then explain why data leakage exists for collaborative filtering models. Through carefully designed experiments, we show that all models indeed recommend future items that are not available at the time point of a test instance, as the result of data leakage. The experiments are conducted with four widely used baseline models - BPR, NeuMF, SASRec, and LightGCN, on four popular offline datasets - MovieLens-25M, Yelp, Amazon-music, and Amazon-electronic, adopting leave-last-one-out data split. We further show that data leakage does impact models’ recommendation accuracy. Their relative performance orders thus become unpredictable with different amount of leaked future data in training. To evaluate recommendation systems in a realistic manner in offline setting, we propose a timeline scheme, which calls for a revisit of the recommendation model design.

Time-Aware Explainable Recommendation via Updating Enabled Online Prediction

“…Therefore, the recommender results and explanations of our proposed online prediction method will be more convincing. The impact of data leakage is recognized by existing works [ 14 , 15 ], but we are the first to offer a comprehensive critical study on this issue under the explainable recommendation scenario.…”

“…Data ordering refers to arranging the interactions randomly or by a timestamp and splitting focus on the ratio of training and test sets. More recently, data leakage issue caused by not observing global timeline in recommender system has attracted increasing attention [ 15 , 29 ]. The damage of data leakage is disastrous, rendering the recommendations invalid, since future interactions are used to predict current user preference.…”

“…Therefore, future knowledge that is not expected to be available at the time of recommendation is utilized, thus data leakage occurs [ 29 ]. The impact of data leakage on recommendation results is widely studied, which causes high accuracy in cross validation but poor result on new data [ 13 , 15 ].…”

“…Therefore, knowledge should not be known at the time of training is utilized, that is, data leakage occurs. As a result, data leakage leads to unrealistic high accuracy and unreasonable explainability, which cannot translate directly into good performance in a real world production scenario [ 15 ].…”

Progress in recommender systems research: Crisis? What crisis?

“…An equally problematic issue was recently reported in the analysis by Sun et al (2020) who found that 37% of their examined works tuned the hyperparameters of their new method on the test set. If such a procedure was admissible, it is trivial to come up with an algorithm for implicit feedback that uses a parameter for each user‐item combination and then “learns” the best setting for this variable by looking at the test set.…”