GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Wang, Kexin; Nandan, Thakur,; Reimers, Nils; Gurevych, Iryna

doi:10.48550/arxiv.2112.07577

Cited by 12 publications

(18 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For example, the model is trained on Web search data and is transferred to medical domains. It has drawn considerable attention from the neural IR community [23,39,45,46]. To provide a deeper understanding of extrapolation evaluation, we investigate how benchmark extrapolation results correlate domain transfer ability (RQ3).…”

Section: Relationship With Transfer Abilitymentioning

confidence: 99%

Evaluating Interpolation and Extrapolation Performance of Neural Retrieval Models

Zhan¹,

Xie²,

Mao³

et al. 2022

Preprint

View full text Add to dashboard Cite

A retrieval model should not only interpolate the training data but also extrapolate well to the queries that are rather different from the training data. While dense retrieval (DR) models have been demonstrated to achieve better retrieval performance than the traditional term-based retrieval models, we still know little about whether they can extrapolate. To shed light on the research question, we investigate how DR models perform in both the interpolation and extrapolation regimes. We first investigate the distribution of training and test data on popular retrieval benchmarks and identify a considerable overlap in query entities, query intent, and relevance labels. This finding implies that the performance on these test sets is biased towards interpolation and cannot accurately reflect the extrapolation capacity. Therefore, to evaluate the extrapolation performance of DR models, we propose two resampling strategies for existing retrieval benchmarks and comprehensively investigate how DR models perform. Results show that DR models may interpolate as well as complex interaction-based models (e.g., BERT and ColBERT) but extrapolate substantially worse. Among various DR training strategies, text-encoding pretraining and target-domain pretraining are particularly effective for improving the extrapolation capacity. Finally, we compare the extrapolation capacity with domain transfer ability. Despite its simplicity and ease of use, the extrapolation performance can reflect the domain transfer ability in some domains of the BEIR dataset, further highlighting the feasibility of our approaches in evaluating the generalizability of DR models.

show abstract

Section: Relationship With Transfer Abilitymentioning

confidence: 99%

Evaluating Interpolation and Extrapolation Performance of Neural Retrieval Models

Zhan¹,

Xie²,

Mao³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Focusing on improving the transfer learning effectiveness of dense retrievers, Ma et al (2021) and Wang et al (2021) use supervised sequence-to-sequence models to augment the training data. They generate questions from texts from different collections and use these synthetic question-text pairs as positive training examples.…”

Section: Related Workmentioning

confidence: 99%

InPars: Data Augmentation for Information Retrieval using Large Language Models

Bonifacio¹,

Abonizio²,

Fadaee³

et al. 2022

Preprint

View full text Add to dashboard Cite

The information retrieval community has recently witnessed a revolution due to large pretrained transformer models. Another key ingredient for this revolution was the MS MARCO dataset, whose scale and diversity has enabled zero-shot transfer learning to various tasks. However, not all IR tasks and domains can benefit from one single dataset equally. Extensive research in various NLP tasks has shown that using domain-specific training data, as opposed to a general-purpose one, improves the performance of neural models (Yu et al., 2021;Sharami et al., 2022). In this work, we harness the fewshot capabilities of large pretrained language models as synthetic data generators for IR tasks. We show that models finetuned solely on our unsupervised dataset outperform strong baselines such as BM25 as well as recently proposed selfsupervised dense retrieval methods. Furthermore, retrievers finetuned on both supervised and our synthetic data achieve better zero-shot transfer than models finetuned only on supervised data. Code, models, and data are available at https: //github.com/zetaalphavector/inpars

show abstract

“…For a fair comparison, Table 6 only includes models that use the same baseline training strategy as ours. Thus, we exclude approaches that depend on other models for expansion [25,33,51], costly training techniques such as knowledge distillation [9,17,18,38,41,44], or special pretraining [11,20,34] (see Table 8 for more comparisons).…”

Section: Evaluation Of Single Model Fusionmentioning

confidence: 99%

“…Even considering a general method such as knowledge distillation (KD), there are many different strategies. For example, we use a lightweight ColBERT teacher, while GPL [44], ColBERTv2 [41], and SPLADEv2 [9] use a more expensive cross-encoder teacher. First, a comparison with Contriever [20], GPL [44], and GTR [34], in columns (e)-(i): We observe that the generalization capability of different retrievers can be improved by increasing the model size.…”

Section: Evaluation Of Single Model Fusionmentioning

confidence: 99%

“…For example, we use a lightweight ColBERT teacher, while GPL [44], ColBERTv2 [41], and SPLADEv2 [9] use a more expensive cross-encoder teacher. First, a comparison with Contriever [20], GPL [44], and GTR [34], in columns (e)-(i): We observe that the generalization capability of different retrievers can be improved by increasing the model size. However, models with more expressive capacity (i.e., ColBERTv2 and SPLADEv2) show equally good generalization capability but sacrifice retrieval efficiency, either in terms of index size or retrieval latency.…”

Section: Evaluation Of Single Model Fusionmentioning

confidence: 99%

See 1 more Smart Citation

A Dense Representation Framework for Lexical and Semantic Matching

Lin¹,

Lin²

2022

Preprint

View full text Add to dashboard Cite

Lexical and semantic matching capture different successful approaches to text retrieval and the fusion of their results has proven to be more effective and robust than either alone. Prior work performs hybrid retrieval by conducting lexical and semantic text matching using different systems (e.g., Lucene and Faiss, respectively) and then fusing their model outputs. In contrast, our work integrates lexical representations with dense semantic representations by densifying high-dimensional lexical representations into what we call low-dimensional dense lexical representations (DLRs). Our experiments show that DLRs can effectively approximate the original lexical representations, preserving effectiveness while improving query latency. Furthermore, we can combine dense lexical and semantic representations to generate dense hybrid representations (DHRs) that are more flexible and yield faster retrieval compared to existing hybrid techniques. Finally, we explore jointly training lexical and semantic representations in a single model and empirically show that the resulting DHRs are able to combine the advantages of each individual component. Our best DHR model is competitive with state-of-the-art single-vector and multi-vector dense retrievers in both in-domain and zero-shot evaluation settings. Furthermore, our model is both faster and requires smaller indexes, making our dense representation framework an attractive approach to text retrieval. Our code is available at https://github.com/castorini/dhr.

show abstract

GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Cited by 12 publications

References 20 publications

Evaluating Interpolation and Extrapolation Performance of Neural Retrieval Models

Evaluating Interpolation and Extrapolation Performance of Neural Retrieval Models

InPars: Data Augmentation for Information Retrieval using Large Language Models

A Dense Representation Framework for Lexical and Semantic Matching

Contact Info

Product

Resources

About