CVTT: Cross-Validation Through Time

Abstract

The practical aspects of evaluating recommender systems is an actively discussed topic in the research community. While many current evaluation techniques bring performance down to a single-value metric as a straightforward approach for model comparison, it is based on a strong assumption of the methods' stable performance over time. In this paper, we argue that leaving out a method's continuous performance can lead to losing valuable insight into joint data-method effects. We propose the Cross-Validation Thought Time (CVTT) technique to perform more detailed evaluations, which focus on model cross-validation performance over time. Using the proposed technique, we conduct a detailed analysis of popular RecSys algorithms' performance against various metrics and datasets.We also compare several data preparation and evaluation strategies to analyze their impact on model performance. Our results show that model performance can vary significantly over time, and both data and evaluation setup can have a marked effect on it.

Impact of temporal context on recommender systems along global timeline

“…To illustrate the performance of DIL over time, we use the example of LightGCN on the Amazon-books dataset and plot the Recall@20 scores over the four testing periods in Figure 5.4. It is observed that the ranking orders of the baselines vary across different periods, which aligns with findings in prior studies such as [18] on other datasets. This observation emphasizes the importance of monitoring evaluation results over time to avoid potential bias in a particular period.…”

“…Some studies have stressed the necessity of considering "time" factor in offline evaluation of recommender system [18][19][20]163]. The authors of [163] argue that a setting which strictly follow a global timeline is a more realistic setting for evaluation.…”

“…They suggest the tuning of time window for more reflective recommendation performance for each recommendation model. In [20] and [18] also consider the impact of temporal factors on recommendation performance, demonstrating that performance varies over time as more interactions are observed. They argue that this variability suggests that context changes over time in recommendations.…”

“…To achieve this objective, we analyse the data splitting strategies in offline evaluation, which have garnered considerable interests from other researchers [15][16][17][18][19][20][21][22].…”