Estimating Position Bias without Intrusive Interventions

Abstract

Presentation bias is one of the key challenges when learning from implicit feedback in search engines, as it confounds the relevance signal. While it was recently shown how counterfactual learningto-rank (LTR) approaches [18] can provably overcome presentation bias when observation propensities are known, it remains to show how to effectively estimate these propensities. In this paper, we propose the first method for producing consistent propensity estimates without manual relevance judgments, disruptive interventions, or restrictive relevance modeling assumptions. First, we show how to harvest a specific type of intervention data from historic feedback logs of multiple different ranking functions, and show that this data is sufficient for consistent propensity estimation in the position-based model. Second, we propose a new extremum estimator that makes effective use of this data. In an empirical evaluation, we find that the new estimator provides superior propensity estimates in two real-world systems -Arxiv Full-text Search and Google Drive Search. Beyond these two points, we find that the method is robust to a wide range of settings in simulation studies.

Fairness in Recommendation Ranking through Pairwise Comparisons

“…Building real-world recommenders face a variety of challenges. Two that relate to the challenges in fairness are the temporal dynamics [33,48,26,9] and biased training data [29,15,3]. These issues do not just make training difficult but also evaluation of recommender performance [42].…”

“…Part of the challenge in studying fairness in recommender systems is that they are complex. They often consist of multiple models [47,24], must balance multiple goals [36,50], and are difficult to evaluate due to extreme and skewed sparsity [8] and numerous dynamics [33,3]. All of these issues are hardly resolved in the recommender system community, and present additional challenges in improving recommender fairness.…”

“…What a user was interested in yesterday they may not be interested in tomorrow, and we only know a user's preferences if we recommend them an item. As a result, metrics are often biased (in the statistical sense) by the previous recommender system [3], and while a large body of research works to do unbiased offline evaluation [43,42], this is very difficult due to the large item space, extreme sparsity of feedback, and evolving users and items. These issues only become more salient when trying to measure recommender system fairness, and even more so when trying to evaluate complete rankings.…”

FairRec: Two-Sided Fairness for Personalized Recommendations in Two-Sided Platforms

“…This being the first work on two-sided-fair-recommendation posed as a fair-allocation problem, we focused on a basic setting without position bias[3], where customers pay more attention to the top ranked products than the lower ranked ones.…”

“…An allocation is said to satisfy envy-freeness if the bundle of items allocated to each agent is as valuable to her as the bundle allocated to any other agent[26,48,50] 3. An allocation is said to satisfy proportional-fair-share if each agent receives a bundle of value at least 1/|U | t h of her total value for all the items[47].FairRec: Two-Sided Fairness for Personalized Recommendations in Two-Sided Platforms WWW '20, April 20-24, 2020, Taipei, Taiwan…”