Learning from User Interactions in Personal Search via Attribute Parameterization

Bendersky, Michael; Wang, Xuanhui; Metzler, Donald; Najork, Marc

doi:10.1145/3018661.3018712

Cited by 36 publications

(42 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As the aim of our work is to compare counterfactual and online LTR approaches, we consider propensity estimation beyond the scope of this paper and assume the propensity scores are known a priori. This is a reasonable assumption, as practitioners typically first perform a randomization experiment to measure the observation probabilities before applying a counterfactual learning algorithm [4].…”

Section: Propensity Estimation Methodsmentioning

confidence: 99%

“…Furthermore, gathering interactions is much less costly than expert annotations [5,20]. Additionally, unlike LTR from annotated datasets, LTR from user interactions can respect privacysensitive settings [4]. However, a big disadvantage of user interactions is that they often contain different types of bias and noise.…”

Section: Introductionmentioning

confidence: 99%

“…However, a big disadvantage of user interactions is that they often contain different types of bias and noise. Hence, LTR methods that learn from user interactions mainly focus on removing bias from the learning process [4,16,24].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

To Model or to Intervene

Jagerman

Oosterhuis

Rijke

2019

Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Learning to Rank (LTR) from user interactions is challenging as user feedback often contains high levels of bias and noise. At the moment, two methodologies for dealing with bias prevail in the field of LTR: counterfactual methods that learn from historical data and model user behavior to deal with biases; and online methods that perform interventions to deal with bias but use no explicit user models. For practitioners the decision between either methodology is very important because of its direct impact on end users. Nevertheless, there has never been a direct comparison between these two approaches to unbiased LTR. In this study we provide the first benchmarking of both counterfactual and online LTR methods under different experimental conditions. Our results show that the choice between the methodologies is consequential and depends on the presence of selection bias, and the degree of position bias and interaction noise. In settings with little bias or noise counterfactual methods can obtain the highest ranking performance; however, in other circumstances their optimization can be detrimental to the user experience. Conversely, online methods are very robust to bias and noise but require control over the displayed rankings. Our findings confirm and contradict existing expectations on the impact of model-based and intervention-based methods in LTR, and allow practitioners to make an informed decision between the two methodologies. CCS CONCEPTS• Information systems → Learning to rank.

show abstract

Section: Propensity Estimation Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

To Model or to Intervene

Jagerman

Oosterhuis

Rijke

2019

Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…User interaction data such as clicks is another important signal for learning-to-rank models in email search. Bendersky et al [4] leveraged user interactions by attribute parameterization. Wang et al [48] mitigated the position bias in click data for better training of the model.…”

Section: Email Searchmentioning

confidence: 99%

Separate and Attend in Personal Email Search

Karimzadehgan

Zhuang

et al. 2020

Proceedings of the 13th International Conference on Web Search and Data Mining

Self Cite

View full text Add to dashboard Cite

In personal email search, user queries often impose different requirements on different aspects of the retrieved emails. For example, the query "my recent flight to the US" requires emails to be ranked based on both textual contents and recency of the email documents, while other queries such as "medical history" do not impose any constraints on the recency of the email. Recent deep learning-to-rank models for personal email search often directly concatenate dense numerical features 1 (e.g., document age) with embedded sparse features (e.g., n-gram embeddings). In this paper, we first show with a set of experiments on synthetic datasets that direct concatenation of dense and sparse features does not lead to the optimal search performance of deep neural ranking models. To effectively incorporate both sparse and dense email features into personal email search ranking, we propose a novel neural model, SepAttn. SepAttn first builds two separate neural models to learn from sparse and dense features respectively, and then applies an attention mechanism at the prediction level to derive the final prediction from these two models. We conduct a comprehensive set of experiments on a large-scale email search dataset, and demonstrate that our SepAttn model consistently improves the search quality over the baseline models.

show abstract

“…To overcome the challenges that neither labeled queries nor aggregated clicks are available in email search, we apply an unsupervised hierarchical clustering algorithm based on truncated SVD [12] and varimax rotation [21] to obtain coarse-to-fine query types. The key idea is that queries of the same type will share the same or similar query attributes [2]. Therefore, we can obtain query types by clustering queries based on these attributes.…”

Section: Introductionmentioning

confidence: 99%

Multi-Task Learning for Email Search Ranking with Auxiliary Query Clustering

Shen

Karimzadehgan

Bendersky

et al. 2018

Proceedings of the 27th ACM International Conference on Information and Knowledge Management

Self Cite

View full text Add to dashboard Cite

User information needs vary significantly across different tasks, and therefore their queries will also differ considerably in their expressiveness and semantics. Many studies have been proposed to model such query diversity by obtaining query types and building query-dependent ranking models. These studies typically require either a labeled query dataset or clicks from multiple users aggregated over the same document. These techniques, however, are not applicable when manual query labeling is not viable, and aggregated clicks are unavailable due to the private nature of the document collection, e.g., in email search scenarios. In this paper, we study how to obtain query type in an unsupervised fashion and how to incorporate this information into query-dependent ranking models. We first develop a hierarchical clustering algorithm based on truncated SVD and varimax rotation to obtain coarse-to-fine query types. Then, we study three querydependent ranking models, including two neural models that leverage query type information as additional features, and one novel multi-task neural model that views query type as the label for the auxiliary query cluster prediction task. This multi-task model is trained to simultaneously rank documents and predict query types. Our experiments on tens of millions of real-world email search queries demonstrate that the proposed multi-task model can significantly outperform the baseline neural ranking models, which either do not incorporate query type information or just simply feed query type as an additional feature.

show abstract

Learning from User Interactions in Personal Search via Attribute Parameterization

Cited by 36 publications

References 29 publications

To Model or to Intervene

To Model or to Intervene

Separate and Attend in Personal Email Search

Multi-Task Learning for Email Search Ranking with Auxiliary Query Clustering

Contact Info

Product

Resources

About