Neural-IR-Explorer: A Content-Focused Tool to Explore Neural Re-ranking Results

Hofstätter, Sebastian; Zlabinger, Markus; Hanbury, Allan

doi:10.1007/978-3-030-45442-5_58

Cited by 8 publications

(8 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They found the first passage to be a very strong indicator of overall relevance. Hofstätter et al [14] created the Neural IR-Explorer to help us better understand single word-interactions in the score aggregation of neural re-ranking models.…”

Section: Related Workmentioning

confidence: 99%

Fine-Grained Relevance Annotations for Multi-Task Document Ranking and Question Answering

Hofstätter

Zlabinger

Sertkan

et al. 2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

Self Cite

View full text Add to dashboard Cite

There are many existing retrieval and question answering datasets. However, most of them either focus on ranked list evaluation or single-candidate question answering. This divide makes it challenging to properly evaluate approaches concerned with ranking documents and providing snippets or answers for a given query. In this work, we present FiRA: a novel dataset of Fine-Grained Relevance Annotations. We extend the ranked retrieval annotations of the Deep Learning track of TREC 2019 with passage and word level graded relevance annotations for all relevant documents. We use our newly created data to study the distribution of relevance in long documents, as well as the attention of annotators to specific positions of the text. As an example, we evaluate the recently introduced TKL document ranking model. We find that although TKL exhibits state-of-the-art retrieval results for long documents, it misses many relevant passages.

show abstract

Section: Related Workmentioning

confidence: 99%

Fine-Grained Relevance Annotations for Multi-Task Document Ranking and Question Answering

Hofstätter

Zlabinger

Sertkan

et al. 2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

Self Cite

View full text Add to dashboard Cite

show abstract

“…The max-sum operator scans the matrix of all term-by-term interactions, which is a technique inspired by earlier works on kernelpooling [18,56]. The term-by-term interaction matrix creates transparency in the scoring, as it allows to inspect the source of different scoring parts, while being mapped to human-readable word units [19]. However, the usefulness of this feature is reduced by the use of special tokens, especially by the query expansion with MASK tokens, as it is non-trivial to explain reliably to the users what each MASK token stands for.…”

Section: Colbert Architecturementioning

confidence: 99%

Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction

Hofstätter¹,

Khattab²,

Althammer³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Recent progress in neural information retrieval has demonstrated large gains in effectiveness, while often sacrificing the efficiency and interpretability of the neural model compared to classical approaches. This paper proposes ColBERTer, a neural retrieval model using contextualized late interaction (ColBERT) with enhanced reduction. Along the effectiveness Pareto frontier, ColBERTer's reductions dramatically lower ColBERT's storage requirements while simultaneously improving the interpretability of its token-matching scores. To this end, ColBERTer fuses single-vector retrieval, multivector refinement, and optional lexical matching components into one model. For its multi-vector component, ColBERTer reduces the number of stored vectors per document by learning unique whole-word representations for the terms in each document and learning to identify and remove word representations that are not essential to effective scoring. We employ an explicit multi-task, multi-stage training to facilitate using very small vector dimensions. Results on the MS MARCO and TREC-DL collection show that ColBERTer can reduce the storage footprint by up to 2.5×, while maintaining effectiveness. With just one dimension per token in its smallest setting, ColBERTer achieves index storage parity with the plaintext size, with very strong effectiveness results. Finally, we demonstrate ColBERTer's robustness on seven high-quality outof-domain collections, yielding statistically significant gains over traditional retrieval baselines.does doxycycline contain sulfa BERT tokenized (9 subword-tokens): 'does', 'do', '##xy', '##cy', '##cl', '##ine', 'contain', 'sul', '##fa'ColBERTer BOW 2 (30 saved vectors from 84 subword-tokens): photosensitivity doxycycline 12.9 sulfa 14.2 sunburned rash clothing sunlight allergic compound drugs containing 6.6 take safely wear . is no 4.7 exposed ... Fulltext: No doxycycline is not a sulfa containing compound, so you may take it safely if you are allergic to sulfa drugs. You should be aware, however, that doxycycline may cause photosensitivity, so you should wear appropriate clothing, or you may get easily sunburned or develop a rash if you are exposed to sunlight.

show abstract

“…We conducted the annotation campaign using the FiRA interface [16,17] and ran the campaign for 7 days with a fixed deadline 2 . To control the quality of the judgements during the campaign, we monitored the average number of judgements per 12 hours and we observe the daily average annotation time per relevance grade to detect random judgements.…”

Section: Annotation Campaignmentioning

confidence: 99%

TripJudge: A Relevance Judgement Test Collection for TripClick Health Retrieval

Althammer,

Hofstätter,

Verberne

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Robust test collections are crucial for Information Retrieval research. Recently there is a growing interest in evaluating retrieval systems for domain-specific retrieval tasks, however these tasks often lack a reliable test collection with human-annotated relevance assessments following the Cranfield paradigm. In the medical domain, the TripClick collection was recently proposed, which contains click log data from the Trip search engine and includes two click-based test sets. However the clicks are biased to the retrieval model used, which remains unknown, and a previous study shows that the test sets have a low judgement coverage for the Top-10 results of lexical and neural retrieval models. In this paper we present the novel, relevance judgement test collection TripJudge for TripClick health retrieval. We collect relevance judgements in an annotation campaign and ensure the quality and reusability of TripJudge by a variety of ranking methods for pool creation, by multiple judgements per query-document pair and by an at least moderate inter-annotator agreement. We compare system evaluation with TripJudge and TripClick and find that that click and judgement-based evaluation can lead to substantially different system rankings. CCS CONCEPTS• Information systems → Test collections.

show abstract

Neural-IR-Explorer: A Content-Focused Tool to Explore Neural Re-ranking Results

Cited by 8 publications

References 12 publications

Fine-Grained Relevance Annotations for Multi-Task Document Ranking and Question Answering

Fine-Grained Relevance Annotations for Multi-Task Document Ranking and Question Answering

Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction

TripJudge: A Relevance Judgement Test Collection for TripClick Health Retrieval

Contact Info

Product

Resources

About