Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models

Nair, Suraj; Yang, Eugene; Lawrie, Dawn; Duh, Kevin; McNamee, Paul; Murray, Kenton; Mayfield, James; Oard, Douglas W.

doi:10.1007/978-3-030-99736-6_26

Cited by 19 publications

(13 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Background and Related Workmentioning

confidence: 99%

“…Recently, dense retrieval models have been adapted to CLIR by replacing the encoder with a multilingual pretrained model, such as mBERT, XLM or XLM-R [3,27,32]. To utilize existing monolingual collections with a large number of relevance labels such as MS MARCO [2], dense retrievers with multilingual embeddings can be trained on such corpora with zero-shot transfer to CLIR by leveraging the multilinguality of the encoder [28,32]. Alternatively, with the help of translation models, one can translate the monolingual training collection into the language pair of interest and train the retriever on it (a "translate-train" approach) [32,38].…”

Section: Background and Related Workmentioning

confidence: 99%

“…To utilize existing monolingual collections with a large number of relevance labels such as MS MARCO [2], dense retrievers with multilingual embeddings can be trained on such corpora with zero-shot transfer to CLIR by leveraging the multilinguality of the encoder [28,32]. Alternatively, with the help of translation models, one can translate the monolingual training collection into the language pair of interest and train the retriever on it (a "translate-train" approach) [32,38]. This training scheme encourages the model to bring the representations of related queries and documents closer across languages.…”

Section: Background and Related Workmentioning

confidence: 99%

“…(e.g, BERT [11]) with a retrieval objective using a large number of query-document pairs such as the ones from MS MARCO [2] or Natural Questions [25]. Recent work showed that these models are effective for CLIR when trained with monolingual query-document pairs, enabling zero-shot transfer [28,32,37]. Alternatively, training the model with translated MS MARCO (translate-train) is more effective but also much more expensive [32,39].…”

mentioning

confidence: 99%

“…Recent work showed that these models are effective for CLIR when trained with monolingual query-document pairs, enabling zero-shot transfer [28,32,37]. Alternatively, training the model with translated MS MARCO (translate-train) is more effective but also much more expensive [32,39].…”

mentioning

confidence: 99%

See 4 more Smart Citations

C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval

Yang,

Nair,

Chandradevan

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Pretrained language models have improved effectiveness on numerous tasks, including ad-hoc retrieval. Recent work has shown that continuing to pretrain a language model with auxiliary objectives before fine-tuning on the retrieval task can further improve retrieval effectiveness. Unlike monolingual retrieval, designing an appropriate auxiliary task for cross-language mappings is challenging. To address this challenge, we use comparable Wikipedia articles in different languages to further pretrain off-the-shelf multilingual pretrained models before fine-tuning on the retrieval task. We show that our approach yields improvements in retrieval effectiveness. CCS CONCEPTS• Information systems → Retrieval models and ranking.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Section: Background and Related Workmentioning

confidence: 99%

Section: Background and Related Workmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval

Yang,

Nair,

Chandradevan

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

HC4: A New Suite of Test Collections for Ad Hoc CLIR

Lawrie

Mayfield

Oard

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

A Unified Framework for Learned Sparse Retrieval

Nguyen

MacAvaney

Yates

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Learned sparse retrieval (LSR) is a family of first-stage retrieval methods that are trained to generate sparse lexical representations of queries and documents for use with an inverted index. Many LSR methods have been recently introduced, with Splade models achieving state-of-the-art performance on MSMarco. Despite similarities in their model architectures, many LSR methods show substantial differences in effectiveness and efficiency. Differences in the experimental setups and configurations used make it difficult to compare the methods and derive insights. In this work, we analyze existing LSR methods and identify key components to establish an LSR framework that unifies all LSR methods under the same perspective. We then reproduce all prominent methods using a common codebase and re-train them in the same environment, which allows us to quantify how components of the framework affect effectiveness and efficiency. We find that (1) including document term weighting is most important for a method's effectiveness, (2) including query weighting has a small positive impact, and (3) document expansion and query expansion have a cancellation effect. As a result, we show how removing query expansion from a state-of-the-art model can reduce latency significantly while maintaining effectiveness on MSMarco and TripClick benchmarks. Our code is publicly available. 3

show abstract

Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models

Cited by 19 publications

References 27 publications

C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval

C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval

HC4: A New Suite of Test Collections for Ad Hoc CLIR

A Unified Framework for Learned Sparse Retrieval

Contact Info

Product

Resources

About