TripClick: The Log Files of a Large Health Web Search Engine

Rekabsaz, Navid; Lesota, Oleg; Schedl, Markus; Brassey, Jon; Eickhoff, Carsten

doi:10.1145/3404835.3463242

Cited by 33 publications

(22 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Methodology. We selected seven datasets from the ir_datasets catalogue [37]: Bio medical (TREC Covid [50,52], TripClick [40], NFCorpus [4]), Entity centric (DBPedia Entity [14]), informal language (Antique [13], TREC Podcast [23]), news cables (TREC Robust 04 [49]). The datasets are not based on web collections, have at least 50 queries, and importantly contain judgements from both relevant and non-relevant categories.…”

Section: Out-of-domain Robustnessmentioning

confidence: 99%

Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction

Hofstätter¹,

Khattab²,

Althammer³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent progress in neural information retrieval has demonstrated large gains in effectiveness, while often sacrificing the efficiency and interpretability of the neural model compared to classical approaches. This paper proposes ColBERTer, a neural retrieval model using contextualized late interaction (ColBERT) with enhanced reduction. Along the effectiveness Pareto frontier, ColBERTer's reductions dramatically lower ColBERT's storage requirements while simultaneously improving the interpretability of its token-matching scores. To this end, ColBERTer fuses single-vector retrieval, multivector refinement, and optional lexical matching components into one model. For its multi-vector component, ColBERTer reduces the number of stored vectors per document by learning unique whole-word representations for the terms in each document and learning to identify and remove word representations that are not essential to effective scoring. We employ an explicit multi-task, multi-stage training to facilitate using very small vector dimensions. Results on the MS MARCO and TREC-DL collection show that ColBERTer can reduce the storage footprint by up to 2.5×, while maintaining effectiveness. With just one dimension per token in its smallest setting, ColBERTer achieves index storage parity with the plaintext size, with very strong effectiveness results. Finally, we demonstrate ColBERTer's robustness on seven high-quality outof-domain collections, yielding statistically significant gains over traditional retrieval baselines.does doxycycline contain sulfa BERT tokenized (9 subword-tokens): 'does', 'do', '##xy', '##cy', '##cl', '##ine', 'contain', 'sul', '##fa'ColBERTer BOW 2 (30 saved vectors from 84 subword-tokens): photosensitivity doxycycline 12.9 sulfa 14.2 sunburned rash clothing sunlight allergic compound drugs containing 6.6 take safely wear . is no 4.7 exposed ... Fulltext: No doxycycline is not a sulfa containing compound, so you may take it safely if you are allergic to sulfa drugs. You should be aware, however, that doxycycline may cause photosensitivity, so you should wear appropriate clothing, or you may get easily sunburned or develop a rash if you are exposed to sunlight.

show abstract

Section: Out-of-domain Robustnessmentioning

confidence: 99%

Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction

Hofstätter¹,

Khattab²,

Althammer³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, the Criteo dataset only includes single item recommendations and not slates. Parallel to our work, Rekabsaz et al (2021) released an information retrieval dataset that includes click log entries from a health website. They include the top 20 retrieved documents from each slate, but do not identify how far the user scrolled.…”

Section: Exposure Modeling In Recommender Systemsmentioning

confidence: 99%

Dynamic Slate Recommendation with Gated Recurrent Units and Thompson Sampling

Eide¹,

Leslie²,

Frigessi³

2021

Preprint

View full text Add to dashboard Cite

We consider the problem of recommending relevant content to users of an internet platform in the form of lists of items, called slates. We introduce a variational Bayesian Recurrent Neural Net recommender system that acts on time series of interactions between the internet platform and the user, and which scales to real world industrial situations. The recommender system is tested both online on real users, and on an offline dataset collected from a Norwegian web-based marketplace, FINN.no, that is made public for research. This is one of the first publicly available datasets which includes all the slates that are presented to users as well as which items (if any) in the slates were clicked on. Such a data set allows us to move beyond the common assumption that implicitly assumes that users are considering all possible items at each interaction. Instead we build our likelihood using the items that are actually in the slate, and evaluate the strengths and weaknesses of both approaches theoretically and in experiments. We also introduce a hierarchical prior for the item parameters based on group memberships. Both item parameters and user preferences are learned probabilistically. Furthermore, we combine our model with bandit strategies to ensure learning, and introduce 'in-slate Thompson Sampling' which makes use of the slates to maximise explorative opportunities. We show experimentally that explorative recommender strategies perform on par or above their greedy counterparts. Even without making use of exploration to learn more effectively, click rates increase simply because of improved diversity in the recommended slates.

show abstract

“…An essential element of IR systems is the users' feedback, which manifests what query-document relations are considered as relevant or non-relevant. Such relevance relations are typically achieved either through explicit relevance judgements [1], or implicit relevance estimations deduced from users' interactions [24]. Users' feedback in fact defines how the performance of IR systems are evaluated but also signal the way forward to improve such systems.…”

Section: Introductionmentioning

confidence: 99%

Do Perceived Gender Biases in Retrieval Results Affect Relevance Judgements?

Krieg¹,

Parada-Cabaleiro²,

Schedl³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

This work investigates the effect of gender-stereotypical biases in the content of retrieved results on the relevance judgement of users/annotators. In particular, since relevance in information retrieval (IR) is a multi-dimensional concept, we study whether the value and quality of the retrieved documents for some bias-sensitive queries can be judged differently when the content of the documents represents different genders. To this aim, we conduct a set of experiments where the genders of the participants are known as well as experiments where the participants' genders are not specified. The set of experiments comprise of retrieval tasks, where participants perform a rated relevance judgement for different search query and search result document compilations. The shown documents contain different gender indications and are either relevant or non-relevant to the query. The results show the differences between the average judged relevance scores among documents with various gender contents. Our work initiates further research on the connection of the perception of gender stereotypes in users with their judgements and effects on IR systems, and aim to raise awareness about the possible biases in this domain.

show abstract

TripClick: The Log Files of a Large Health Web Search Engine

Cited by 33 publications

References 24 publications

Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction

Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction

Dynamic Slate Recommendation with Gated Recurrent Units and Thompson Sampling

Do Perceived Gender Biases in Retrieval Results Affect Relevance Judgements?

Contact Info

Product

Resources

About