Pretrained Transformers for Text Ranking: BERT and Beyond

Lin, Jimmy; Nogueira, Rodrigo; Yates, Andrew

doi:10.48550/arxiv.2010.06467

Cited by 62 publications

(65 citation statements)

References 359 publications

(408 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The success of pre-trained transformer-based language models such as BERT [19] and T5 [47] on several IR benchmarks-a comprehensive account of the effectiveness gains can be found in [29]-has lead to research on understanding their behaviour and the reasons behind their significant gains in ranking effectiveness [12,32,43,46,61].…”

Section: Model Understandingmentioning

confidence: 99%

“…We use different ranking models that cover from lexical traditional models (Trad) such as BM25, to neural ranking models (NN) such as KNRM and neural ranking models that employ transformer-based language models (TNN) such as BERT. For all of our experiments, we apply BM25 as a first stage retriever and re-rank the top 100 results with the neural ranking models, which is an established and efficient approach [29].…”

Section: Ranking Modelsmentioning

confidence: 99%

See 1 more Smart Citation

Evaluating the Robustness of Retrieval Pipelines with Query Variation Generators

Penha¹,

Câmara²,

Hauff³

2021

Preprint

View full text Add to dashboard Cite

Heavily pre-trained transformers for language modelling, such as BERT, have shown to be remarkably effective for Information Retrieval (IR) tasks. IR benchmarks evaluate the effectiveness of (neural) ranking models based on the premise that a single query is used to instantiate the underlying information need. However, previous research has shown that (I) queries generated by users for a fixed information need are extremely variable and, in particular, (II) neural models are brittle and often easily make mistakes when tested with adversarial examples, i.e. examples with minimal modifications that do not change its label. Motivated by those observations we aim to answer the following question with our work: how robust are retrieval pipelines with respect to different variations in queries that do not change the queries' semantics? In order to obtain queries that are representative of users' querying variability, we first created a taxonomy based on the manual annotation of transformations occurring in a dataset (specifically UQV100) of user created query variations. For example, from the query 'cures for a bald spot' to the variation 'cures for baldness' we are applying a paraphrasing transformation that replaces words with synonyms. For each syntax-changing category of our taxonomy, we employ different automatic methods that when applied to a query generate a query variation. We conduct experiments on two datasets (TREC-DL-2019 and ANTIQUE) and create a total of 2430 query variations from 243 topics across both datasets. Our experimental results for two different IR tasks reveal that retrieval pipelines are not robust to query variations that maintain the content the same, with effectiveness drops of ∼20% on average when compared with the original query as provided in the datasets. Our findings indicate that further work is required to make retrieval pipelines with neural ranking models more robust and that IR collections should include query variations, e.g. using the methods proposed here, for a single information need to better understand models capabilities. The code and datasets are available at https://github.com/Guzpenha/query_variation_generators.

show abstract

Section: Model Understandingmentioning

confidence: 99%

Section: Ranking Modelsmentioning

confidence: 99%

Evaluating the Robustness of Retrieval Pipelines with Query Variation Generators

Penha¹,

Câmara²,

Hauff³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In this section, we briefly review relevant work in information retrieval, and application of machine learning to this problem. This is not an exhaustive review, and we refer the reader to Manning et al (2008), Mitra et al (2018 and Lin et al (2020) for a more complete introduction to the field.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Dense Information Retrieval with Contrastive Learning

Izacard¹,

Caron²,

Hosseini³

et al. 2021

Preprint

View full text Add to dashboard Cite

Information retrieval is an important component in natural language processing, for knowledge intensive tasks such as question answering and fact checking. Recently, information retrieval has seen the emergence of dense retrievers, based on neural networks, as an alternative to classical sparse methods based on term-frequency. These models have obtained state-of-the-art results on datasets and tasks where large training sets are available. However, they do not transfer well to new domains or applications with no training data, and are often outperformed by term-frequency methods such as BM25 which are not supervised. Thus, a natural question is whether it is possible to train dense retrievers without supervision. In this work, we explore the limits of contrastive learning as a way to train unsupervised dense retrievers, and show that it leads to strong retrieval performance. More precisely, we show on the BEIR benchmark that our model outperforms BM25 on 11 out of 15 datasets. Furthermore, when a few thousands examples are available, we show that fine-tuning our model on these leads to strong improvements compared to BM25. Finally, when used as pre-training before fine-tuning on the MS-MARCO dataset, our technique obtains state-of-the-art results on the BEIR benchmark.

show abstract

“…Our work is orthogonal to the extensive research on neural encoders in that we provide a new Learning to rank (LTR) is a long-established interdisciplinary research area at the intersection of machine learning and information retrieval (Liu, 2009). Neural rankers are dominating in ranking virtually all modalities recently, including text ranking (Lin et al, 2020), image retrieval (Gordo et al, 2016), and tabular data ranking (Qin et al, 2021). Many LTR papers focus on more effective loss functions (Qin et al, 2010;Bruch et al, 2020) to rank items with respect to a query.…”

Section: Related Workmentioning

confidence: 99%

Rank4Class: A Ranking Formulation for Multiclass Classification

Wang¹,

Qin²,

Yan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Multiclass classification (MCC) is a fundamental machine learning problem which aims to classify each instance into one of a predefined set of classes. Given an instance, a classification model computes a score for each class, all of which are then used to sort the classes. The performance of a classification model is usually measured by Top-K Accuracy/Error (e.g., K = 1 or 5). In this paper, we do not aim to propose new neural representation learning models as most recent works do, but to show that it is easy to boost MCC performance with a novel formulation through the lens of ranking. In particular, by viewing MCC as to rank classes for an instance, we first argue that ranking metrics, such as Normalized Discounted Cumulative Gain (NDCG), can be more informative than existing Top-K metrics. We further demonstrate that the dominant neural MCC architecture can be formulated as a neural ranking framework with a specific set of design choices. Based on such generalization, we show that it is straightforward and intuitive to leverage techniques from the rich information retrieval literature to improve the MCC performance out of the box. Extensive empirical results on both text and image classification tasks with diverse datasets and backbone models (e.g., BERT and ResNet for text and image classification) show the value of our proposed framework.

show abstract

Pretrained Transformers for Text Ranking: BERT and Beyond

Cited by 62 publications

References 359 publications

Evaluating the Robustness of Retrieval Pipelines with Query Variation Generators

Evaluating the Robustness of Retrieval Pipelines with Query Variation Generators

Unsupervised Dense Information Retrieval with Contrastive Learning

Rank4Class: A Ranking Formulation for Multiclass Classification

Contact Info

Product

Resources

About