A Re-visit of the Popularity Baseline in Recommender Systems

Abstract

Popularity is often included in experimental evaluation to provide a reference performance for a recommendation task. To understand how popularity baseline is defined and evaluated, we sample 12 papers from top-tier conferences including KDD, WWW, SIGIR, and RecSys, and 6 open source toolkits. We note that the widely adopted MostPop baseline simply ranks items based on the number of interactions in the training data. We argue that the current evaluation of popularity (i) does not reflect the popular items at the time when a user interacts with the system, and (ii) may recommend items released after a user's last interaction with the system. On the widely used MovieLens dataset, we show that the performance of popularity could be significantly improved by 70% or more, if we consider the popular items at the time point when a user interacts with the system. We further show that, on MovieLens dataset, the users having lower tendencies on movies tend to follow the crowd and rate more popular movies. Movie lovers who rate a large number of movies, rate movies based on their own preferences and interests. Through this study, we call for a re-visit of the popularity baseline in recommender system to better reflect its effectiveness.

It Is Different When Items Are Older

“…Positivity bias leads to an over-representation of positive feedback because users rate the items they like more often [37]. The effect of these biases is generally dynamic: they can change drastically over time [9,26,58]. For instance, items are rarely popular for very extended periods of time, and therefore, we may expect a dynamic effect between the age of items and popularity bias.…”

“…While the impact of dynamic bias has previously been pointed out [26,58], no prior debiasing method considers a scenario in which both selection bias and user preferences change over time. All existing debiased recommendation methods assume a static effect of selection bias regardless of whether they model dynamic user preferences.…”

“…In practice, selection bias is usually dynamic, not static [9,26]. Typically, the popularity of an item changes with item-age [8,26], i.e., the time since its publication.…”

“…In practice, selection bias is usually dynamic, not static [9,26]. Typically, the popularity of an item changes with item-age [8,26], i.e., the time since its publication. Figure 1 shows the number of ratings items received as they get older in the MovieLens dataset (red line).…”

PP-Rec: News Recommendation with Personalized User Interest and Time-aware News Popularity

“…We compare PP-Rec with two groups of baselines. The first group is popularity-based news recommendation methods, including: (1) ViewNum (Yang, 2016): using the number of news view to measure news popularity; (2) RecentPop (Ji et al, 2020): using the number of news view in recent time to measure news popularity; (3) SCENE (Li et al, 2011): using view frequency to measure news popularity and adjusting the ranking of news with same topics based on their popularity; (4) CTR (Ji et al, 2020): using news CTR to measure news popularity. The second group is personalized news recommendation methods, containing: (1) EBNR (Okura et al, 2017): utilizing an auto-encoder to learn news representations and a GRU network to learn user representations; (2) DKN (Wang et al, 2018): utilizing a knowledge-aware CNN network to learn news representations from news titles and entities;…”