Evaluating and Analyzing Click Simulation in Web Search

Malkevich, Stepan; Markov, Ilia; Michailova, Elena; Rijke, Maarten de

doi:10.1145/3121050.3121096

Cited by 5 publications

(4 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, most click models are designed to optimize the likelihood of observed clicks. Although Malkevich et al [20] have shown that UBM is better than DBN in terms of click simulation, the ranker trained with DBN performed better than the ranker trained with UBM in our experiments. This suggests that optimizing click likelihood doesn't necessarily produce the best bias correction model for unbiased learning to rank.…”

Section: Comparison With Click Modelscontrasting

confidence: 78%

Unbiased Learning to Rank with Unbiased Propensity Estimation

Luo

et al. 2018

The 41st International ACM SIGIR Conference on Research &Amp; Development in Information Retrieval

156

237

View full text Add to dashboard Cite

Learning to rank with biased click data is a well-known challenge. A variety of methods has been explored to debias click data for learning to rank such as click models, result interleaving and, more recently, the unbiased learning-to-rank framework based on inverse propensity weighting. Despite their differences, most existing studies separate the estimation of click bias (namely the propensity model) from the learning of ranking algorithms. To estimate click propensities, they either conduct online result randomization, which can negatively affect the user experience, or offline parameter estimation, which has special requirements for click data and is optimized for objectives (e.g. click likelihood) that are not directly related to the ranking performance of the system. In this work, we address those problems by unifying the learning of propensity models and ranking models. We find that the problem of estimating a propensity model from click data is a dual problem of unbiased learning to rank. Based on this observation, we propose a Dual Learning Algorithm (DLA) that jointly learns an unbiased ranker and an unbiased propensity model. DLA is an automatic unbiased learning-to-rank framework as it directly learns unbiased ranking models from biased click data without any preprocessing. It can adapt to the change of bias distributions and is applicable to online learning. Our empirical experiments with synthetic and realworld data show that the models trained with DLA significantly outperformed the unbiased learning-to-rank algorithms based on result randomization and the models trained with relevance signals extracted by click models.

show abstract

Section: Comparison With Click Modelscontrasting

confidence: 78%

Unbiased Learning to Rank with Unbiased Propensity Estimation

Luo

et al. 2018

The 41st International ACM SIGIR Conference on Research &Amp; Development in Information Retrieval

156

237

View full text Add to dashboard Cite

show abstract

“…The quality of click models is often evaluated by the Log-Likelihood and Perplexity [61], but also other reliability measures exist [62]. In previous work, click models have mainly been evaluated on semi-public web search datasets, e.g., from Yahoo!…”

Section: Click Modelsmentioning

confidence: 99%

“…3.4.1 Log-Likelihood. This is a standard evaluation measure of click models, and it was found that better scores correlate with a higher fidelity of simulated clicks [61]. We determine it over a run with |Q| queries and ranking length as follows:…”

Section: Evaluation Measuresmentioning

confidence: 99%

Validating Synthetic Usage Data in Living Lab Environments

Breuer,

Fuhr,

Schaer

2024

J. Data and Information Quality

View full text Add to dashboard Cite

Evaluating retrieval performance without editorial relevance judgments is challenging, but instead, user interactions can be used as relevance signals. Living labs offer a way for small-scale platforms to validate information retrieval systems with real users. If enough user interaction data is available, click models can be parameterized from historical sessions to evaluate systems before exposing users to experimental rankings. However, interaction data is sparse in living labs, and little is studied about how click models can be validated for reliable user simulations when click data is available in moderate amounts. This work introduces an evaluation approach for validating synthetic usage data generated by click models in data-sparse human-in-the-loop environments like living labs. We ground our methodology on the click model’s estimates about a system ranking compared to a reference ranking for which the relative performance is known. Our experiments compare different click models and their reliability and robustness as more session log data becomes available. In our setup, simple click models can reliably determine the relative system performance with already 20 logged sessions for 50 queries. In contrast, more complex click models require more session data for reliable estimates, but they are a better choice in simulated interleaving experiments when enough session data is available. While it is easier for click models to distinguish between more diverse systems, it is harder to reproduce the system ranking based on the same retrieval algorithm with different interpolation weights. Our setup is entirely open, and we share the code to reproduce the experiments.

show abstract

“…With a small compromise on the bottom of long ranked lists, MMF achieves superior performance and significantly outperforms the state-of-the-art fairness algorithms not only in top-k relevance, but also in top-k fairness. As most people would examine or only examine the top results on a result page [8,13,19,27], MMF is highly competitive and preferable in real LTR systems and applications.…”

Section: Introductionmentioning

confidence: 99%

Maximizing Marginal Fairness for Dynamic Learning to Rank

Yang

2021

Proceedings of the Web Conference 2021

View full text Add to dashboard Cite

Rankings, especially those in search and recommendation systems, often determine how people access information and how information is exposed to people. Therefore, how to balance the relevance and fairness of information exposure is considered as one of the key problems for modern IR systems. As conventional ranking frameworks that myopically sorts documents with their relevance will inevitably introduce unfair result exposure, recent studies on ranking fairness mostly focus on dynamic ranking paradigms where result rankings can be adapted in real-time to support fairness in groups (i.e., races, genders, etc.). Existing studies on fairness in dynamic learning to rank, however, often achieve the overall fairness of document exposure in ranked lists by significantly sacrificing the performance of result relevance and fairness on the top results.To address this problem, we propose a fair and unbiased ranking method named Maximal Marginal Fairness (MMF). The algorithm integrates unbiased estimators for both relevance and merit-based fairness while providing an explicit controller that balances the selection of documents to maximize the marginal relevance and fairness in top-k results. Theoretical and empirical analysis shows that, with small compromises on long list fairness, our method achieves superior efficiency and effectiveness comparing to the state-of-the-art algorithms in both relevance and fairness for top-k rankings. CCS CONCEPTS• Information systems → Learning to rank.

show abstract

Evaluating and Analyzing Click Simulation in Web Search

Cited by 5 publications

References 20 publications

Unbiased Learning to Rank with Unbiased Propensity Estimation

Unbiased Learning to Rank with Unbiased Propensity Estimation

Validating Synthetic Usage Data in Living Lab Environments

Maximizing Marginal Fairness for Dynamic Learning to Rank

Contact Info

Product

Resources

About