Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval

Yilmaz, Zeynep Akkalyoncu; Yang, Wei; Zhang, Haotian; Lin, Jimmy

doi:10.18653/v1/d19-1352

Cited by 113 publications

(141 citation statements)

References 23 publications

Supporting

Mentioning

138

Contrasting

Order By: Relevance

“…The T5-3B results in bold are significantly better (p < 0.05) than T5-large, T5-base, and the corresponding baseline (BM25 or BM25+RM3), based on the Student's paired t-test with Bonferroni corrections. We compare our model with Birch (Yilmaz et al, 2019), BERT-MaxP (Dai and Callan, 2019), and PARADE , which are BERT-based models that represent the state of the art. BERT-MaxP and PARADE results are from fine-tuning on the MS MARCO data and then fine-tuning again on Robust04 (via cross-validation).…”

Section: Resultsmentioning

confidence: 99%

“…This leads to the standard multi-stage pipeline architecture where first-stage retrieval is followed by reranking using one or more machine learning models (Asadi and Lin, 2013;Nogueira et al, 2019a). This architecture underlies nearly all transformerbased approaches to document retrieval today, for example, CEDR (MacAvaney et al, 2019), BERT-MaxP (Dai and Callan, 2019), Birch (Yilmaz et al, 2019), and PARADE .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Document Ranking with a Pretrained Sequence-to-Sequence Model

Nogueira¹,

Jiang²,

Lin³

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

Self Cite

259

145

View full text Add to dashboard Cite

This work proposes the use of a pretrained sequence-to-sequence model for document ranking. Our approach is fundamentally different from a commonly adopted classificationbased formulation based on encoder-only pretrained transformer architectures such as BERT. We show how a sequence-to-sequence model can be trained to generate relevance labels as "target tokens", and how the underlying logits of these target tokens can be interpreted as relevance probabilities for ranking. Experimental results on the MS MARCO passage ranking task show that our ranking approach is superior to strong encoderonly models. On three other document retrieval test collections, we demonstrate a zeroshot transfer-based approach that outperforms previous state-of-the-art models requiring indomain cross-validation. Furthermore, we find that our approach significantly outperforms an encoder-only architecture in a data-poor setting. We investigate this observation in more detail by varying target tokens to probe the model's use of latent knowledge. Surprisingly, we find that the choice of target tokens impacts effectiveness, even for words that are closely related semantically. This finding sheds some light on why our sequence-to-sequence formulation for document ranking is effective. Code and models are available at pygaggle.ai.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Document Ranking with a Pretrained Sequence-to-Sequence Model

Nogueira¹,

Jiang²,

Lin³

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

Self Cite

259

145

View full text Add to dashboard Cite

show abstract

“…We identified numerous works on neural models published in recent years that would benefit from an evaluation and analysis based on FiRA. A common approach to utilizing the large-scale pre-trained BERT model [8] in document ranking is to apply BERT to passages [30], overlapping windows [32], or single sentences [1]. In all these cases, BERT produces partial results that need to be aggregated externally to produce a final ranking score that could be compared with traditional full-document judgements.…”

Section: Related Workmentioning

confidence: 99%

Fine-Grained Relevance Annotations for Multi-Task Document Ranking and Question Answering

Hofstätter

Zlabinger

Sertkan

et al. 2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

There are many existing retrieval and question answering datasets. However, most of them either focus on ranked list evaluation or single-candidate question answering. This divide makes it challenging to properly evaluate approaches concerned with ranking documents and providing snippets or answers for a given query. In this work, we present FiRA: a novel dataset of Fine-Grained Relevance Annotations. We extend the ranked retrieval annotations of the Deep Learning track of TREC 2019 with passage and word level graded relevance annotations for all relevant documents. We use our newly created data to study the distribution of relevance in long documents, as well as the attention of annotators to specific positions of the text. As an example, we evaluate the recently introduced TKL document ranking model. We find that although TKL exhibits state-of-the-art retrieval results for long documents, it misses many relevant passages.

show abstract

“…Following this observation, our system's architecture consists of modules ("IR primitives") that declare dependencies on other modules. 2 For example, in Figure 1, Searcher depends on an Index (which depends on a Collection), and Reranker depends on a Trainer and Extractor. Dependencies may specify both a module type (e.g., Searcher) and a default module class (e.g., BM25), which can be overridden by the user via the configuration.…”

Section: Architecturementioning

confidence: 99%

“…Each method makes a different efficiency vs. effectiveness trade-off and potentially operates on different features or document representations. With the growing popularity of computationally expensive BERT-based models (e.g., [2,6,13,17]) and substantially more expensive models based on T5 [18], the telescoping approach becomes particularly appealing as a means for reducing the number of documents these models evaluate.…”

Section: Introductionmentioning

confidence: 99%

Flexible IR Pipelines with Capreolus

Yates

Jose

Zhang

et al. 2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

Self Cite

View full text Add to dashboard Cite

While a number of recent open-source toolkits for training and using neural information retrieval models have greatly simplified experiments with neural reranking methods, they essentially hard code a "search-then-rerank" experimental pipeline. These pipelines consist of an efficient first-stage ranking method, like BM25, followed by a neural reranking method. Deviations from this setup often require hacks; some improvements, like adding a second reranking step that uses a more expensive neural method, are infeasible without major code changes. In order to improve the flexibility of such toolkits, we propose implementing experimental pipelines as dependency graphs of functional "IR primitives, " which we call modules, that can be used and combined as needed. For example, a neural IR pipeline may rerank results from a Searcher module that efficiently retrieves results from an Index module that it depends on. In turn, the Index depends on a Collection to index, which is provided by the pipeline. This Searcher module is self-contained: the pipeline does not need to know about or interact with the Index of the Searcher, which is transparently shared among Searcher modules when possible (e.g., a BM25 and a QL Searcher might share the same Index). Similarly, a Reranker module might depend on a Trainer (e.g., Tensorflow), feature Extractor, Tokenizer, etc. In both cases, the pipeline needs to interact only with the Reranker or Searcher directly; the complexity of their dependencies is hidden and intelligently managed. We rewrite the Capreolus toolkit to take this approach and demonstrate its use.

show abstract

Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval

Cited by 113 publications

References 23 publications

Document Ranking with a Pretrained Sequence-to-Sequence Model

Document Ranking with a Pretrained Sequence-to-Sequence Model

Fine-Grained Relevance Annotations for Multi-Task Document Ranking and Question Answering

Flexible IR Pipelines with Capreolus

Contact Info

Product

Resources

About