Unbiased offline recommender evaluation for missing-not-at-random implicit feedback

Yang, Longqi; Cui, Yin; Xuan, Yue; Wang, Chenyang; Belongie, Serge; Estrin, Deborah

doi:10.1145/3240323.3240355

Cited by 185 publications

(167 citation statements)

References 19 publications

Supporting

Mentioning

158

Contrasting

Order By: Relevance

“…Causal inference is also used to handle the missing-not-at-random (MNAR) nature [29,44] of user feedback. IPS estimators were used to adjust the item selection bias of explicit feedback [41] and implicit feedback [50]. Another approach to MNAR is exposure modeling [26], which decomposes missing feedback to either a user's unawareness of or dislike for an item.…”

Section: Proposed Sampling Methodsmentioning

confidence: 99%

Uplift-based evaluation and optimization of recommenders

Sato

Singh

Takemori

et al. 2019

Proceedings of the 13th ACM Conference on Recommender Systems

View full text Add to dashboard Cite

Recommender systems aim to increase user actions such as clicks and purchases. Typical evaluations of recommenders regard the purchase of a recommended item as a success. However, the item may have been purchased even without the recommendation. An uplift is defned as an increase in user actions caused by recommendations. Situations with and without a recommendation cannot both be observed for a specifc user-item pair at a given time instance, making uplift-based evaluation and optimization challenging. This paper proposes new evaluation metrics and optimization methods for the uplift in a recommender system. We apply a causal inference framework to estimate the average uplift for the ofine evaluation of recommenders. Our evaluation protocol leverages both purchase and recommendation logs under a currently deployed recommender system, to simulate the cases both with and without recommendations. This enables the ofine evaluation of the uplift for newly generated recommendation lists. For optimization, we need to defne positive and negative samples that are specifc to an uplift-based approach. For this purpose, we deduce four classes of items by observing purchase and recommendation logs. We derive the relative priorities among these four classes in terms of the uplift and use them to construct both pointwise and pairwise sampling methods for uplift optimization. Through dedicated experiments with three public datasets, we demonstrate the efectiveness of our optimization methods in improving the uplift. CCS CONCEPTS • Information systems → Recommender systems; • Computing methodologies → Learning from implicit feedback.

show abstract

Section: Proposed Sampling Methodsmentioning

confidence: 99%

Uplift-based evaluation and optimization of recommenders

Sato

Singh

Takemori

et al. 2019

Proceedings of the 13th ACM Conference on Recommender Systems

View full text Add to dashboard Cite

show abstract

“…In this setting, we study how the user was exposed to the items before providing the ratings and then how this will affect the next predictions. This is equivalent to studying the Missing Not At Random (MNAR) problem [35]. In fact, by studying the distribution of the missing data, we can infer the effect of the bias on the predictions and/or the training.…”

Section: Exposure Biasmentioning

confidence: 99%

Theoretical Modeling of the Iterative Properties of User Discovery in a Collaborative Filtering Recommender System

Khenissi

Boujelbene

Nasraoui

2020

Fourteenth ACM Conference on Recommender Systems

View full text Add to dashboard Cite

The closed feedback loop in recommender systems is a common setting that can lead to different types of biases. Several studies have dealt with these biases by designing methods to mitigate their effect on the recommendations. However, most existing studies do not consider the iterative behavior of the system where the closed feedback loop plays a crucial role in incorporating different biases into several parts of the recommendation steps. We present a theoretical framework to model the asymptotic evolution of the different components of a recommender system operating within a feedback loop setting, and derive theoretical bounds and convergence properties on quantifiable measures of the user discovery and blind spots. We also validate our theoretical findings empirically using a real-life dataset and empirically test the efficiency of a basic exploration strategy within our theoretical framework. Our findings lay the theoretical basis for quantifying the effect of feedback loops and for designing Artificial Intelligence and machine learning algorithms that explicitly incorporate the iterative nature of feedback loops in the machine learning and recommendation process.

show abstract

“…Recommenders are often evaluated and compared offline using datasets collected from online platforms [18]. Evaluation can be done by using prediction accuracy or information retrieval metrics.…”

Section: Related Workmentioning

confidence: 99%

Hybrid Data Set Optimization in Recommender Systems Using Fuzzy T-Norms

Papaleonidas

Pimenidis

Iliadis

2019

IFIP Advances in Information and Communication Technology

View full text Add to dashboard Cite

A recommender system uses specific algorithms and techniques in order to suggest specific services, goods or other type of recommendations that users could be interested in. User's preferences or ratings are used as inputs and top-N recommendations are produced by the system. The evaluation of the recommendations is usually based on accuracy metrics such as the Mean Absolute Error (MAE) and the Root Mean Squared Error (RMSE), while on the other hand Precision and Recall is used to measure the quality of the top-N recommendations. Recommender systems development has been mainly focused in the development of new recommendation algorithms. However, one of the major problems in modern offline recommendation system is the sparsity of the datasets and the selection of the suitable users Y that could produce the best recommendations for users X. In this paper, we propose an algorithm that uses Fuzzy sets and Fuzzy norms in order to evaluate the correlation between users in the data set so the system can select and use only the most relevant users. At the same time, we are extending our previous work about Reproduction of experiments in recommender systems by developing new explanations and variables for the proposed new algorithm. Our proposed approach has been experimentally evaluated using a real dataset and the results show that it is really efficient and it can increase both accuracy and quality of recommendations.

show abstract

Unbiased offline recommender evaluation for missing-not-at-random implicit feedback

Cited by 185 publications

References 19 publications

Uplift-based evaluation and optimization of recommenders

Uplift-based evaluation and optimization of recommenders

Theoretical Modeling of the Iterative Properties of User Discovery in a Collaborative Filtering Recommender System

Hybrid Data Set Optimization in Recommender Systems Using Fuzzy T-Norms

Contact Info

Product

Resources

About