Differentiable Unbiased Online Learning to Rank

Oosterhuis, Harrie; Rijke, Maarten de

doi:10.1145/3269206.3271686

Cited by 74 publications

(119 citation statements)

References 38 publications

Supporting

Mentioning

117

Contrasting

Order By: Relevance

“…Furthermore, to our surprise we find that some properties asserted to pertain to CLTR or OLTR methods in previously published work appear to be lacking when tested. For instance, in contrast with previously published expectations [24] OLTR is not substantially faster at learning than CLTR, and while always assumed to be safe [36], CLTR may be detrimental to the user experience when deployed under high-levels of noise.…”

Section: Introductioncontrasting

confidence: 60%

“…Online Learning to Rank (OLTR) [8,24,29,37] aims to learn by directly interacting with users. OLTR algorithms affect the data gathered during the learning process because they have control over what is displayed to users.…”

Section: Online Learning To Rankmentioning

confidence: 99%

“…Instead, they use stochasticity in the displayed results to handle selection and position biases. In addition, their properties are only based on simple assumptions about user behavior [24,37], e.g., a relevant document is more likely to be clicked than a non-relevant document. Thus, in cases where users are hard to model OLTR may have an advantage over CLTR.…”

Section: Online Learning To Rankmentioning

confidence: 99%

“…All of these extensions improve on the initial learning speed of DBGD. However, no extension has been shown to improve long-term performance [24,26,28]. Moreover, even under ideal circumstances and with a very large number of iterations DBGD is unable to reach levels of performance comparable to LTR from labeled datasets [22,24].…”

Section: Dueling Bandit Gradient Descentmentioning

confidence: 99%

“…In reaction to the shortcomings of DBGD, recent work has introduced the Pairwise Differentiable Gradient Descent (PDGD) algorithm [24]. In contrast to DBGD, PDGD does not depend on sampling model variations or online evaluation methods.…”

Section: Pairwise Differentiable Gradient Descentmentioning

confidence: 99%

See 4 more Smart Citations

To Model or to Intervene

Jagerman

Oosterhuis

Rijke

2019

Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

Learning to Rank (LTR) from user interactions is challenging as user feedback often contains high levels of bias and noise. At the moment, two methodologies for dealing with bias prevail in the field of LTR: counterfactual methods that learn from historical data and model user behavior to deal with biases; and online methods that perform interventions to deal with bias but use no explicit user models. For practitioners the decision between either methodology is very important because of its direct impact on end users. Nevertheless, there has never been a direct comparison between these two approaches to unbiased LTR. In this study we provide the first benchmarking of both counterfactual and online LTR methods under different experimental conditions. Our results show that the choice between the methodologies is consequential and depends on the presence of selection bias, and the degree of position bias and interaction noise. In settings with little bias or noise counterfactual methods can obtain the highest ranking performance; however, in other circumstances their optimization can be detrimental to the user experience. Conversely, online methods are very robust to bias and noise but require control over the displayed rankings. Our findings confirm and contradict existing expectations on the impact of model-based and intervention-based methods in LTR, and allow practitioners to make an informed decision between the two methodologies. CCS CONCEPTS• Information systems → Learning to rank.

show abstract

Section: Introductioncontrasting

confidence: 60%

Section: Online Learning To Rankmentioning

confidence: 99%

Section: Online Learning To Rankmentioning

confidence: 99%

Section: Dueling Bandit Gradient Descentmentioning

confidence: 99%

Section: Pairwise Differentiable Gradient Descentmentioning

confidence: 99%

See 3 more Smart Citations

To Model or to Intervene

Jagerman

Oosterhuis

Rijke

2019

Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

show abstract

Optimizing Ranking Models in an Online Setting

Oosterhuis

Rijke

2019

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Online Learning to Rank (OLTR) methods optimize ranking models by directly interacting with users, which allows them to be very efficient and responsive. All OLTR methods introduced during the past decade have extended on the original OLTR method: Dueling Bandit Gradient Descent (DBGD). Recently, a fundamentally different approach was introduced with the Pairwise Differentiable Gradient Descent (PDGD) algorithm. To date the only comparisons of the two approaches are limited to simulations with cascading click models and low levels of noise. The main outcome so far is that PDGD converges at higher levels of performance and learns considerably faster than DBGD-based methods. However, the PDGD algorithm assumes cascading user behavior, potentially giving it an unfair advantage. Furthermore, the robustness of both methods to high levels of noise has not been investigated. Therefore, it is unclear whether the reported advantages of PDGD over DBGD generalize to different experimental conditions. In this paper, we investigate whether the previous conclusions about the PDGD and DBGD comparison generalize from ideal to worst-case circumstances. We do so in two ways. First, we compare the theoretical properties of PDGD and DBGD, by taking a critical look at previously proven properties in the context of ranking. Second, we estimate an upper and lower bound on the performance of methods by simulating both ideal user behavior and extremely difficult behavior, i.e., almost-random non-cascading user models. Our findings show that the theoretical bounds of DBGD do not apply to any common ranking model and, furthermore, that the performance of DBGD is substantially worse than PDGD in both ideal and worst-case circumstances. These results reproduce previously published findings about the relative performance of PDGD vs. DBGD and generalize them to extremely noisy and non-cascading circumstances.

show abstract

Counterfactual Online Learning to Rank

Zhuang

Zuccon

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Exploiting users' implicit feedback, such as clicks, to learn rankers is attractive as it does not require editorial labelling effort, and adapts to users' changing preferences, among other benefits. However, directly learning a ranker from implicit data is challenging, as users' implicit feedback usually contains bias (e.g., position bias, selection bias) and noise (e.g., clicking on irrelevant but attractive snippets, adversarial clicks). Two main methods have arisen for optimizing rankers based on implicit feedback: counterfactual learning to rank (CLTR), which learns a ranker from the historical click-through data collected from a deployed, logging ranker; and online learning to rank (OLTR), where a ranker is updated by recording user interaction with a result list produced by multiple rankers (usually via interleaving).In this paper, we propose a counterfactual online learning to rank algorithm (COLTR) that combines the key components of both CLTR and OLTR. It does so by replacing the online evaluation required by traditional OLTR methods with the counterfactual evaluation common in CLTR. Compared to traditional OLTR approaches based on interleaving, COLTR can evaluate a large number of candidate rankers in a more efficient manner. Our empirical results show that COLTR significantly outperforms traditional OLTR methods. Furthermore, COLTR can reach the same effectiveness of the current state-of-the-art, under noisy click settings, and has room for future extensions.

show abstract

Differentiable Unbiased Online Learning to Rank

Cited by 74 publications

References 38 publications

To Model or to Intervene

To Model or to Intervene

Optimizing Ranking Models in an Online Setting

Counterfactual Online Learning to Rank

Contact Info

Product

Resources

About