Deconfounding User Satisfaction Estimation from Response Rate Bias

Abstract

Improving user satisfaction is at the forefront of industrial recommender systems. While significant progress has been made by utilizing logged implicit data of user-item interactions (i.e., clicks, dwell/watch time, and other user engagement signals), there has been a recent surge of interest in measuring and modeling user satisfaction, as provided by orthogonal data sources. Such data sources typically originate from responses to user satisfaction surveys, which explicitly ask users to rate their experience with the system and/or specific items they have consumed in the recent past. This data can be valuable for measuring and modeling the degree to which a user has had a satisfactory experience on the recommendation platform, since what users do (engagement) does not always align with what users say they want (satisfaction as measured by surveys). We focus on a large-scale industrial system trained on user survey responses to predict user satisfaction. The predictions of the satisfaction model for each user-item pair, combined with the predictions of the other models (e.g., engagement-focused ones), are fed into the ranking component of a real-world recommender system in deciding items to present to the user. It is therefore imperative that the satisfaction model does an equally good job on imputing user satisfaction across slices of users and items, as it would directly impact which items a user is exposed to. However, the data used for training satisfaction models is biased in that users are more likely to respond to a survey when they will respond that they are more satisfied. When the satisfaction survey responses in slices of data with high response rate follow a different distribution than those with low response rate, response rate becomes a confounding factor for user satisfaction estimation. We find positive correlation between response rate and ratings in a large-scale survey dataset collected in our case study. To address this inherent response rate bias in the satisfaction data, we propose an inverse propensity weighting approach within a multi-task learning framework. We extend a simple feed-forward neural network architecture predicting user satisfaction to a shared-bottom multi-task learning architecture with two tasks: the user satisfaction estimation task, and the response rate estimation task. We Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

Causal Intervention for Leveraging Popularity Bias in Recommendation

“…Some efforts have considered causality in recommendation. The first type of work is on confounding effects [10,13,37,44]. For example, [44] takes the de-confounding technique in linear models to learn real interest influenced by unobserved confounders.…”

“…[10] explores the impact of algorithmic confounding on simulated recommender systems. [13] identifies expose rate as a confounder for user satisfaction estimation and uses IPS to handle the confounding problem. In this work, we introduce a different type of confounder that is brought by item popularity.…”

Addressing Confounding Feature Issue for Causal Recommendation

“…Moreover, its causal mechanism is different from us, e.g., it does not consider confounders. To handle bias issues, the most widely considered method is IPW-based methods 6 [10,12,28,30,47], which adjust the training distribution by reweighting training samples with propensity scores. However, the propensities are difficult to set properly, e.g., causing the high variance issues etc [4,55].…”

Learning Robust Recommender from Noisy Implicit Feedback

“…As to the information retrieval domain, early research [46]- [49] starts from using causal inference to remove various bias in user feedback [50], such as position bias [51] and popularity bias [52]. Recently, more techniques in causal inference have been applied to solve the issues in recommendation [53]- [55], for example, causal intervention [56] and counterfactual inference [6]. However, existing work on causal recommendation has never studied the colliding effect or the causality-based inference strategy.…”