TripJudge

Althammer, Sophia; Hofstätter, Sebastian; Verberne, Suzan; Hanbury, Allan

doi:10.1145/3511808.3557714

Cited by 5 publications

(7 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to strengthen the reasoning behind the hypothesized system rankings, we evaluate them with the help of editorial relevance judgments. For this purpose, we use the previously mentioned TripJudge relevance labels [2]. The results in Figure 3 show that the system-oriented experiment gives evidence to the hypothesized relative orderings of the system performance.…”

Section: 1)mentioning

confidence: 93%

“…By increasing , we deteriorate the ranking results in a systematic but also more subtle way, which better simulates incremental and less invasive changes to an existing search platform in an online experiment. 2 The resulting IRM ranking is defined by…”

Section: Experimental Systemsmentioning

confidence: 99%

“…It contains documents and user interaction logs covering a period of seven years, from 2013 to 2020. It was highlighted that the annotation coverage for the top results is low [40], and recently, topical relevance judgments called TripJudge were introduced [2]. In our experiments, we can only use data logs with information about the entire SERP, which are available from 13th August 2016.…”

Section: Datasetmentioning

confidence: 99%

See 2 more Smart Citations

Validating Synthetic Usage Data in Living Lab Environments

Breuer,

Fuhr,

Schaer

2024

J. Data and Information Quality

View full text Add to dashboard Cite

Evaluating retrieval performance without editorial relevance judgments is challenging, but instead, user interactions can be used as relevance signals. Living labs offer a way for small-scale platforms to validate information retrieval systems with real users. If enough user interaction data is available, click models can be parameterized from historical sessions to evaluate systems before exposing users to experimental rankings. However, interaction data is sparse in living labs, and little is studied about how click models can be validated for reliable user simulations when click data is available in moderate amounts. This work introduces an evaluation approach for validating synthetic usage data generated by click models in data-sparse human-in-the-loop environments like living labs. We ground our methodology on the click model’s estimates about a system ranking compared to a reference ranking for which the relative performance is known. Our experiments compare different click models and their reliability and robustness as more session log data becomes available. In our setup, simple click models can reliably determine the relative system performance with already 20 logged sessions for 50 queries. In contrast, more complex click models require more session data for reliable estimates, but they are a better choice in simulated interleaving experiments when enough session data is available. While it is easier for click models to distinguish between more diverse systems, it is harder to reproduce the system ranking based on the same retrieval algorithm with different interpolation weights. Our setup is entirely open, and we share the code to reproduce the experiments.

show abstract

Section: 1)mentioning

confidence: 93%

Section: Experimental Systemsmentioning

confidence: 99%

Section: Datasetmentioning

confidence: 99%

See 1 more Smart Citation

Validating Synthetic Usage Data in Living Lab Environments

Breuer,

Fuhr,

Schaer

2024

J. Data and Information Quality

View full text Add to dashboard Cite

show abstract

“…In this paper, (1) we investigate how the amount of labelled data used for fine-tuning a PLM ranker impacts its effectiveness, (2) we adapt active learning (AL) strategies to the task of training PLM rankers, (3) we propose a budget-aware evaluation schema including aspects of annotation and computation cost, (4) we conduct an extensive analysis of AL strategies for training PLM rankers investigating the trade-offs between effectiveness, annotation budget and computational budget. We do this in the context of three common PLM ranker architectures: cross-encoders (MonoBERT [44]), single representation bi-encoders (DPR [33]) and multi-representation bi-encoders (ColBERT [34]), and two scenarios: ➊ Scratch: the PLM is pre-trained on a background corpus, but has yet to be fine-tuned to the target ranking task and dataset; ➋ Re-Train: domain adaptation of the PLM ranker is performed.…”

Section: Introductionmentioning

confidence: 95%

“…Data annotation typically requires a large manual effort and thus is expensive, especially in domain-specific tasks where annotators should be domain experts. In real-life settings, annotation and computational budget 1 is often limited, especially for start-ups or in domain-specific contexts.…”

Section: Introductionmentioning

confidence: 99%

Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection

Althammer,

Zuccon,

Hofstätter

et al. 2023

Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia P

Self Cite

View full text Add to dashboard Cite

Search methods based on Pretrained Language Models (PLM) have demonstrated great effectiveness gains compared to statistical and early neural ranking models. However, fine-tuning PLM-based rankers requires a great amount of annotated training data. Annotating data involves a large manual effort and thus is expensive, especially in domain specific tasks. In this paper we investigate finetuning PLM-based rankers under limited training data and budget. We investigate two scenarios: fine-tuning a ranker from scratch, and domain adaptation starting with a ranker already fine-tuned on general data, and continuing fine-tuning on a target dataset.We observe a great variability in effectiveness when fine-tuning on different randomly selected subsets of training data. This suggests that it is possible to achieve effectiveness gains by actively selecting a subset of the training data that has the most positive effect on the rankers. This way, it would be possible to fine-tune effective PLM rankers at a reduced annotation budget. To investigate this, we adapt existing Active Learning (AL) strategies to the task of fine-tuning PLM rankers and investigate their effectiveness, also considering annotation and computational costs. Our extensive analysis shows that AL strategies do not significantly outperform random selection of training subsets in terms of effectiveness. We further find that gains provided by AL strategies come at the expense of more assessments (thus higher annotation costs) and AL strategies underperform random selection when comparing effectiveness given a fixed annotation cost. Our results highlight that "optimal" subsets of training data that provide high effectiveness at low annotation cost do exist, but current mainstream AL strategies applied to PLM rankers are not capable of identifying them.

show abstract