Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction

Ma, Xinyu; Guo, Jiafeng; Zhang, Ruqing; Fan, Yixing; Cheng, Xueqi

doi:10.1145/3477495.3531772

Cited by 18 publications

(6 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In experiments, we conduct extensive evaluations of our method on the two long-document retrieval benchmark datasets: MS-Marco Doc (Nguyen et al 2016) and TREC Deep Learning 2019 document retrieval (TREC 2019) (Craswell et al 2020). Following previous works (Ma et al 2022), we use official metrics MRR@100 and Recall@100 (R@100) to report evaluation results on MS-Marco dev, while using nDCG@10 and Recall@100 for TREC 2019.…”

Section: Methodsmentioning

confidence: 99%

“…In particular, to produce consistent embeddings, previous methods directly apply mean-pooling over contextual embeddings for different granularity, which however becomes inferior when the document length goes extremely long and has proven less effective in our pilot experiments. This is the reason why most previous document retrieval works rely on [CLS] embedding paradigm (Ma et al 2022;Xiong et al 2021;Zhan et al 2021b).…”

Section: Global-consistent Granularity Embeddingmentioning

confidence: 99%

“…Following previous works (Ma et al 2022), we detail our pre-training and fine-tuning pipelines (shown in Figure 3) for document retrieval by FGD.…”

Section: Pre-training and Fine-tuning Pipelinementioning

confidence: 99%

“…Stage-0: Pre-training. Initialing a model by selfsupervised pre-training has been proven effective by numerous works (Xiong et al 2021;Zhan et al 2021b;Ma et al 2021Ma et al , 2022, which can be categorized into two groups, i.e., general pre-training and corpus-aware pre-training. Specifically, the former is referred to as PLMs that are pre-trained on general corpora by language modeling (e.g., RoBERTa (Liu et al 2019)).…”

Section: Pre-training and Fine-tuning Pipelinementioning

confidence: 99%

See 3 more Smart Citations

Fine-Grained Distillation for Long Document Retrieval

Zhou,

Shen,

Geng

et al. 2024

AAAI

View full text Add to dashboard Cite

Long document retrieval aims to fetch query-relevant documents from a large-scale collection, where knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. However, in contrast to passages or sentences, retrieval on long documents suffers from the \textit{scope hypothesis} that a long document may cover multiple topics. This maximizes their structure heterogeneity and poses a granular-mismatch issue, leading to an inferior distillation efficacy. In this work, we propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers. While preserving the conventional dense retrieval paradigm, it first produces global-consistent representations crossing different fine granularity and then applies multi-granular aligned distillation merely during training. In experiments, we evaluate our framework on two long-document retrieval benchmarks, which show state-of-the-art performance.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Global-consistent Granularity Embeddingmentioning

confidence: 99%

“…Following previous works (Ma et al 2022), we detail our pre-training and fine-tuning pipelines (shown in Figure 3) for document retrieval by FGD.…”

Section: Pre-training and Fine-tuning Pipelinementioning

confidence: 99%

Section: Pre-training and Fine-tuning Pipelinementioning

confidence: 99%

See 2 more Smart Citations

Fine-Grained Distillation for Long Document Retrieval

Zhou,

Shen,

Geng

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Pre-trained Bi-encoder. Researchers have explored pre-training models for retrieval with the bi-encoder architecture [6,7,14,18,20,30]. For example, Gao and Callan [6] added extra head layers atop the Transformer, with shortcut connections between early outputs and the head, enhancing the CLS embedding of the encoder.…”

Section: Related Workmentioning

confidence: 99%

A Multi-Granularity-Aware Aspect Learning Model for Multi-Aspect Dense Retrieval

Sun,

Bi,

Guo

et al. 2024

Proceedings of the 17th ACM International Conference on Web Search and Data Mining

Self Cite

View full text Add to dashboard Cite

Dense retrieval methods have been mostly focused on unstructured text and less attention has been drawn to structured data with various aspects, e.g., products with aspects such as category and brand. Recent work has proposed two approaches to incorporate the aspect information into item representations for effective retrieval by predicting the values associated with the item aspects. Despite their efficacy, they treat the values as isolated classes (e.g., "Smart Homes", "Home, Garden & Tools", and "Beauty & Health") and ignore their fine-grained semantic relation. Furthermore, they either enforce the learning of aspects into the CLS token, which could confuse it from its designated use for representing the entire content semantics, or learn extra aspect embeddings only with the value prediction objective, which could be insufficient especially when there are no annotated values for an item aspect.Aware of these limitations, we propose a MUlti-granulaRityaware Aspect Learning model (MURAL) for multi-aspect dense retrieval. It leverages aspect information across various granularities to capture both coarse and fine-grained semantic relations between values. Moreover, MURAL incorporates separate aspect embeddings as input to transformer encoders so that the masked language model objective can assist implicit aspect learning even without aspect-value annotations. Extensive experiments on two real-world datasets of products and mini-programs show that MU-RAL outperforms state-of-the-art baselines significantly. Code will be available at the URL 1 .

show abstract