Active learning strategies for rating elicitation in collaborative filtering

Abstract

The accuracy of collaborative-filtering recommender systems largely depends on three factors: the quality of the rating prediction algorithm, and the quantity and quality of available ratings. While research in the field of recommender systems often concentrates on improving prediction algorithms, even the best algorithms will fail if they are fed poor-quality data during training, that is, garbage in, garbage out. Active learning aims to remedy this problem by focusing on obtaining better-quality data that more aptly reflects a user's preferences. However, traditional evaluation of active learning strategies has two major flaws, which have significant negative ramifications on accurately evaluating the system's performance (prediction error, precision, and quantity of elicited ratings). (1) Performance has been evaluated for each user independently (ignoring system-wide improvements). (2) Active learning strategies have been evaluated in isolation from unsolicited user ratings (natural acquisition). In this article we show that an elicited rating has effects across the system, so a typical user-centric evaluation which ignores any changes of rating prediction of other users also ignores these cumulative effects, which may be more influential on the performance of the system as a whole (system centric). We propose a new evaluation methodology and use it to evaluate some novel and state-of-the-art rating elicitation strategies. We found that the system-wide effectiveness of a rating elicitation strategy depends on the stage of the rating elicitation process, and on the evaluation measures (MAE, NDCG, and Precision). In particular, we show that using some common user-centric strategies may actually degrade the overall performance of a system. Finally, we show that the performance of many common active learning strategies changes significantly when evaluated concurrently with the natural acquisition of ratings in recommender systems.

Personality-Based Active Learning for Collaborative Filtering Recommender Systems

“…Active Learning in RSs aims at actively acquiring user preference data to improve the output of the RS [21,7,6]. In [17] six techniques that collaborative filtering systems can use to learn about new users in the sign up process are introduced: entropy where items with the largest rating entropy are preferred; random; popularity; log(popularity) * entropy where items that are both popular and have diverse ratings are preferred; and finally item-item personalized, where the items are proposed randomly until one rating is acquired, then a recommender is used to predict the items that the user is likely to have seen.…”

“…Several approaches have been recently proposed to deal with this problem [20] but the most direct way is to rely on active learning (AL), i.e., to use an initial data acquisition and learning phase. Here the system actively asks the user to rate a set of items, which are identified using a strategy aimed at best revealing the user's interests and consequently at improving the quality of the recommendations [21,7].…”

Alleviating the new user problem in collaborative filtering by exploiting personality information

“…However, regardless of the specific variant that is used, CF methods have a common limitation: the so called new user cold-start problem, which occurs when a system cannot generate personalized and relevant recommendations for a user who has just registered into the system. Although many solutions have been proposed [23,24,33,56,58,72,47,69], this problem is still challenging, and there is not a unique solution for it that can be applied to any domain or situation. Indeed, as we shall show later, different approaches better suit specific situations, e.g., when the new user has entered either zero or only a few ratings.…”

“…In general, in AL it has been shown that asking a user to provide ratings for a set of selected items can improve the accuracy of CF [65,23,21,22,57,58]. Traditional active learning methods need some pre-existent ratings in order to select even more ratings to collect from the user.…”

“…Our method identifies the items to request the user to rate by considering the user personality, which may be easier to obtain than a bootstrapping set of ratings [5], and, since personality is not domain dependent, can be obtained once and then reused in several recommendation domains. Some existing AL approaches have already exploited user personality information [19,5,23]. In contrast to previous work, our personality-based active learning method is optimized for positive-only feedback (e.g., likes, click-through data, item consumption counts) instead of ratings, which is more easily acquired by the system.…”