MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders Are Better Dense Retrievers

Zhou, Kun; Liu, Xiao; Gong, Yeyun; Zhao, Wayne Xin; Jiang, Daxin; Duan, Nan; Wen, Ji-Rong

doi:10.1007/978-3-031-43415-0_37

Cited by 2 publications

(1 citation statement)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, we focus on dense retrieval (DR) that learn low-dimensional semantic embeddings for both queries and documents and then measure the embedding similarities as the relevance scores. To implement DR, a PLM-based dual-encoder has been widely adopted (Karpukhin et al, 2020;Xiong et al, 2020;Zhou et al, 2023) by learning two separate encoders for relevance matching. Compared with sparse retrievers (e.g., BM25), DR models highly rely on large-scale high-quality training data to achieve good performance Ren et al, 2021a;Zhou et al, 2022).…”

Section: Zero-shot Dense Retrievalmentioning

confidence: 99%

A Thorough Examination on Zero-shot Dense Retrieval

Ren,

Qu,

Liu

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Recent years have witnessed the significant advance in dense retrieval (DR) based on powerful pre-trained language models (PLM). DR models have achieved excellent performance in several benchmark datasets, while they are shown to be not as competitive as traditional sparse retrieval models (e.g., BM25) in a zeroshot retrieval setting. However, in the related literature, there still lacks a detailed and comprehensive study on zero-shot retrieval. In this paper, we present the first thorough examination of the zero-shot capability of DR models. We aim to identify the key factors and analyze how they affect zero-shot retrieval performance. In particular, we discuss the effect of several key factors related to source training set, analyze the potential bias from the target dataset, and review and compare existing zeroshot DR models. Our findings provide important evidence to better understand and develop zero-shot DR models.

show abstract

Section: Zero-shot Dense Retrievalmentioning

confidence: 99%