Learning to Rank with Selection Bias in Personal Search

Wang, Xuanhui; Bendersky, Michael; Metzler, Donald; Najork, Marc

doi:10.1145/2911451.2911537

Cited by 245 publications

(272 citation statements)

References 34 publications

Supporting

Mentioning

272

Contrasting

Order By: Relevance

“…Recent work in CLTR has focused on estimating propensities from data [2,35,36]. As the aim of our work is to compare counterfactual and online LTR approaches, we consider propensity estimation beyond the scope of this paper and assume the propensity scores are known a priori.…”

Section: Propensity Estimation Methodsmentioning

confidence: 99%

To Model or to Intervene

Jagerman

Oosterhuis

Rijke

2019

Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Learning to Rank (LTR) from user interactions is challenging as user feedback often contains high levels of bias and noise. At the moment, two methodologies for dealing with bias prevail in the field of LTR: counterfactual methods that learn from historical data and model user behavior to deal with biases; and online methods that perform interventions to deal with bias but use no explicit user models. For practitioners the decision between either methodology is very important because of its direct impact on end users. Nevertheless, there has never been a direct comparison between these two approaches to unbiased LTR. In this study we provide the first benchmarking of both counterfactual and online LTR methods under different experimental conditions. Our results show that the choice between the methodologies is consequential and depends on the presence of selection bias, and the degree of position bias and interaction noise. In settings with little bias or noise counterfactual methods can obtain the highest ranking performance; however, in other circumstances their optimization can be detrimental to the user experience. Conversely, online methods are very robust to bias and noise but require control over the displayed rankings. Our findings confirm and contradict existing expectations on the impact of model-based and intervention-based methods in LTR, and allow practitioners to make an informed decision between the two methodologies. CCS CONCEPTS• Information systems → Learning to rank.

show abstract

Section: Propensity Estimation Methodsmentioning

confidence: 99%

To Model or to Intervene

Jagerman

Oosterhuis

Rijke

2019

Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…Enterprise search is also closely related to personal search (e.g., email search), as both deal with searching in private or access controlled corpora [2,8,12,20,26,39,44]. Even though some success has been found using time-based approaches for personal search [12], relevance-based ranking arising from learning-to-rank deep neural network models has become increasingly popular [39,46] as the sizes of private corpora increase [20].…”

Section: Enterprise Searchmentioning

confidence: 99%

“…Model performance is evaluated using weighted mean reciprocal rank (WMRR), as proposed in [44]. The weighted MRR (WMRR) is calculated using the one clicked document of any query as:…”

Section: Model Evaluationmentioning

confidence: 99%

Domain Adaptation for Enterprise Email Search

Tran

Karimzadehgan

Pasumarthi

et al. 2019

Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

In the enterprise email search setting, the same search engine often powers multiple enterprises from various industries: technology, education, manufacturing, etc. However, using the same global ranking model across different enterprises may result in suboptimal search quality, due to the corpora differences and distinct information needs. On the other hand, training an individual ranking model for each enterprise may be infeasible, especially for smaller institutions with limited data. To address this data challenge, in this paper we propose a domain adaptation approach that fine-tunes the global model to each individual enterprise. In particular, we propose a novel application of the Maximum Mean Discrepancy (MMD) approach to information retrieval, which attempts to bridge the gap between the global data distribution and the data distribution for a given individual enterprise. We conduct a comprehensive set of experiments on a large-scale email search engine, and demonstrate that the MMD approach consistently improves the search quality for multiple individual domains, both in comparison to the global ranking model, as well as several competitive domain adaptation baselines including adversarial learning methods.

show abstract

“…Compared to other methods, IPW is the most applicable approach for our setting, and its assumptions are satisfied in our data. Two recent studies [Schnabel et al 2016;Wang et al 2016] also adopted this method in different machine learning tasks. These studies were published while this article was under review.…”

Section: Correcting Selection Bias Through Ipwmentioning

confidence: 99%

Deriving User Preferences of Mobile Apps from Their Management Activities

Liu

et al. 2017

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

App marketplaces host millions of mobile apps that are downloaded billions of times. Investigating how people manage mobile apps in their everyday lives creates a unique opportunity to understand the behavior and preferences of mobile device users, infer the quality of apps, and improve user experience. Existing literature provides very limited knowledge about app management activities, due to the lack of app usage data at scale. This article takes the initiative to analyze a very large app management log collected through a leading Android app marketplace. The dataset covers 5 months of detailed downloading, updating, and uninstallation activities, which involve 17 million anonymized users and 1 million apps. We present a surprising finding that the metrics commonly used to rank apps in app stores do not truly reflect the users' real attitudes. We then identify behavioral patterns from the app management activities that more accurately indicate user preferences of an app even when no explicit rating is available. A systematic statistical analysis is designed to evaluate machine learning models that are trained to predict user preferences using these behavioral patterns, which features an inverse probability weighting method to correct the selection biases in the training process.

show abstract

Learning to Rank with Selection Bias in Personal Search

Cited by 245 publications

References 34 publications

To Model or to Intervene

To Model or to Intervene

Domain Adaptation for Enterprise Email Search

Deriving User Preferences of Mobile Apps from Their Management Activities

Contact Info

Product

Resources

About