Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking

Oosterhuis, Harrie; Rijke, Maarten de

doi:10.1145/3409256.3409820

Cited by 17 publications

(15 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They suggested that, from a theoretical standpoint, counterfactual LTR exploits position bias better, however they also indicated that empirical results have shown that OLTR (and in particular PDGD) is more reliable. On the other hand, recent works have been focusing on adapting the offline counterfactual learning to the online setting [6,64,34,35]. These works have suggested that OLTR algorithms can benefit from the counterfactual learning framework.…”

Section: Online Learning To Rankmentioning

confidence: 99%

Reinforcement Online Learning to Rank with Unbiased Reward Shaping

Zhuang¹,

Qiao²,

Zuccon³

2022

Preprint

View full text Add to dashboard Cite

Online learning to rank (OLTR) aims to learn a ranker directly from implicit feedback derived from users' interactions, such as clicks. Clicks however are a biased signal: specifically, top-ranked documents are likely to attract more clicks than documents down the ranking (position bias). In this paper, we propose a novel learning algorithm for OLTR that uses reinforcement learning to optimize rankers: Reinforcement Online Learning to Rank (ROLTR). In ROLTR, the gradients of the ranker are estimated based on the rewards assigned to clicked and unclicked documents. In order to de-bias the users' position bias contained in the reward signals, we introduce unbiased reward shaping functions that exploit inverse propensity scoring for clicked and unclicked documents. The fact that our method can also model unclicked documents provides a further advantage in that less users interactions are required to effectively train a ranker, thus providing gains in efficiency. Empirical evaluation on standard OLTR datasets shows that ROLTR achieves state-ofthe-art performance, and provides significantly better user experience than other OLTR approaches. To facilitate the reproducibility of our experiments, we make all experiment code available at https://github.com/ielab/OLTR.

show abstract

Section: Online Learning To Rankmentioning

confidence: 99%

Reinforcement Online Learning to Rank with Unbiased Reward Shaping

Zhuang¹,

Qiao²,

Zuccon³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…While all of these methods can be useful in practice, none optimize rankings metrics directly in a computationally feasible manner. Interestingly, multiple lines of previous work have found PLranking models to be very effective for various ranking tasks: for result randomization in interleaving [16], multileaving [27] and counterfactual evaluation [22]; for exploration in online LTR [21,23]; for fair distributions of attention exposure [11,28]; and for topic diversity in ranking [32,35]. In particular, Bruch et al [3] argue that the stochastic nature of PL models results in more robust ranking performance.…”

Section: Related Workmentioning

confidence: 99%

“…ranking metrics but that it is generally infeasible to compute, existing work has thus approximated this gradient [11,22,23,28] (see Section 4). The computational costs are particularly relevant because the PL model is often used in online settings where it optimization is performed repeatedly and frequently [22,23,28]. For instance, Oosterhuis and de Rijke [23] show that frequently optimizing the logging-policy model during the gathering of data greatly reduces the data-requirements for online/counterfactual LTR.…”

Section: Related Workmentioning

confidence: 99%

“…However, calculating the gradient of a PL ranking model requires an iteration over every possible ranking that the model could produce, i.e., every possible permutation. In practice this computational infeasibility is circumvented by estimating the gradient based on rankings sampled from the model [11,22,23,28]. The main downside of this approach is that it can be computationally very costly.…”

mentioning

confidence: 99%

“…The main downside of this approach is that it can be computationally very costly. This is a particular problem in online settings where optimization is performed periodically as more data is gathered [20,22,23,28].…”

mentioning

confidence: 99%

See 2 more Smart Citations

Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness

Oosterhuis

2021

Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

Recent work has proposed stochastic Plackett-Luce (PL) ranking models as a robust choice for optimizing relevance and fairness metrics. Unlike their deterministic counterparts that require heuristic optimization algorithms, PL models are fully differentiable. Theoretically, they can be used to optimize ranking metrics via stochastic gradient descent. However, in practice, the computation of the gradient is infeasible because it requires one to iterate over all possible permutations of items. Consequently, actual applications rely on approximating the gradient via sampling techniques.In this paper, we introduce a novel algorithm: PL-Rank, that estimates the gradient of a PL ranking model w.r.t. both relevance and fairness metrics. Unlike existing approaches that are based on policy gradients, PL-Rank makes use of the specific structure of PL models and ranking metrics. Our experimental analysis shows that PL-Rank has a greater sample-efficiency and is computationally less costly than existing policy gradients, resulting in faster convergence at higher performance. PL-Rank further enables the industry to apply PL models for more relevant and fairer real-world ranking systems. CCS CONCEPTS• Information systems → Learning to rank.

show abstract