BERT-QE: Contextualized Query Expansion for Document Re-ranking

Zheng, Zhi; Hui, Kai Lung; He, Ben; Han, Xianpei; Sun, Le; Yates, Andrew

doi:10.18653/v1/2020.findings-emnlp.424

Cited by 56 publications

(47 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However the use of BERT models directly within the pseudo-relevance feedback mechanism has seen comparatively little use in the literature. The current approaches leveraging the BERT contextualised embeddings for PRF are Neural PRF [18], BERT-QE [37] and CEQE [23].…”

Section: Related Workmentioning

confidence: 99%

“…feedback chunks that are extracted from the top-ranked feedback documents. This results in an expensive application of many BERT computations -approximately 11× as many GPU operations than a simple BERT reranker [37]. Both Neural PRF and BERT-QE approaches leverage contextualised language models to rerank an initial ranking of documents retrieved by a preliminary sparse retrieval system.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval

Wang

Macdonald

Tonellotto

et al. 2021

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval

View full text Add to dashboard Cite

Pseudo-relevance feedback mechanisms, from Rocchio to the relevance models, have shown the usefulness of expanding and reweighting the users' initial queries using information occurring in an initial set of retrieved documents, known as the pseudo-relevant set. Recently, dense retrieval -through the use of neural contextual language models such as BERT for analysing the documents' and queries' contents and computing their relevance scores -has shown a promising performance on several information retrieval tasks still relying on the traditional inverted index for identifying documents relevant to a query. Two different dense retrieval families have emerged: the use of single embedded representations for each passage and query (e.g. using BERT's [CLS] token), or via multiple representations (e.g. using an embedding for each token of the query and document). In this work, we conduct the first study into the potential for multiple representation dense retrieval to be enhanced using pseudo-relevance feedback. In particular, based on the pseudo-relevant set of documents identified using a first-pass dense retrieval, we extract representative feedback embeddings (using KMeans clustering) -while ensuring that these embeddings discriminate among passages (based on IDF) -which are then added to the query representation. These additional feedback embeddings are shown to both enhance the effectiveness of a reranking as well as an additional dense retrieval operation. Indeed, experiments on the MSMARCO passage ranking dataset show that MAP can be improved by upto 26% on the TREC 2019 query set and 10% on the TREC 2020 query set by the application of our proposed ColBERT-PRF method on a ColBERT dense retrieval approach.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval

Wang

Macdonald

Tonellotto

et al. 2021

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval

View full text Add to dashboard Cite

show abstract

“…An end-to-end neural PRF model (NPRF) proposed by Li et al [14] uses a combination of models to compare document summaries and compute document relevance scores for feedback and achieves limited improvement while only using bag-of-words neural models. Later work combining BERT with a NPRF framework [41] illustrated the importance of an effective first-stage ranking method. A complementary vein of work [23] uses generative approaches to perform document expansion by predicting questions to add to document.…”

Section: Supervised Expansionmentioning

confidence: 99%

CEQE: Contextualized Embeddings for Query Expansion

Naseri

Dalton

Yates

et al. 2021

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

In this work we leverage recent advances in context-sensitive language models to improve the task of query expansion. Contextualized word representation models, such as ELMo and BERT, are rapidly replacing static embedding models. We propose a new model, Contextualized Embeddings for Query Expansion (CEQE), that utilizes queryfocused contextualized embedding vectors. We study the behavior of contextual representations generated for query expansion in ad-hoc document retrieval. We conduct our experiments on probabilistic retrieval models as well as in combination with neural ranking models. We evaluate CEQE on two standard TREC collections: Robust and Deep Learning. We find that CEQE outperforms static embedding-based expansion methods on multiple collections (by up to 18% on Robust and 31% on Deep Learning on average precision) and also improves over proven probabilistic pseudo-relevance feedback (PRF) models. We further find that multiple passes of expansion and reranking result in continued gains in effectiveness with CEQE-based approaches outperforming other approaches. The final model incorporating neural and CEQE-based expansion score achieves gains of up to 5% in P@20 and 2% in AP on Robust over the state-of-the-art transformer-based re-ranking model, Birch.

show abstract

“…Other researchers have also suggested that PRF can use other knowledge, such as Wikipedia. The re-ranking of the PRF results is also carried out by other researchers using machine learning-classification methods [23], BERT [14][15][16], Multi-Stage using History, PRF original, and BERT [24].…”

Section: Related Workmentioning

confidence: 99%

“…BERT is a method widely used in recent research because the improvement of results in performance. However, previous studies used BERT as document ranking method to replace the original PRF that using TFIDF term weighting [14][15][16]. The document results consist other contexts that have semantic similarities and ignore documents that are statistically similar.…”

Section: Introductionmentioning

confidence: 99%

Pseudo-Relevance Feedback Combining Statistical and Semantic Term Extraction for Searching Arabic Documents

Maryamah¹

2021

IJIES

View full text Add to dashboard Cite

Pseudo-relevance feedback (PRF) is an effective query expansion method for searching candidate terms based on top-ranked documents. PRF uses a statistical approach that relies heavily on term-based document retrievals. It causes a semantic gap between query and document that causes vocabulary mismatch problems, such as synonyms and polysemy. In this paper, we proposed pseudo-relevance feedback that combines statistical and semantic term extraction using Term Frequency-Inverse Document Frequency (TFIDF), Bidirectional Encoder Representations from Transformers (BERT), and Yet Another Keyword Extraction (YAKE) for searching Arabic documents. Pseudorelevance feedback will produce top-ranked documents that are relevant to the query. Term extraction on top-ranked documents using TFIDF, BERT, and YAKE was conducted to obtain candidate terms related statistically and semantically, thus understanding the context of the query sentence. The term extraction results will be re-ranked using Borda ranking to ensure that the Top-K candidate terms are relevant and used for searching the final document with the original query. From the experimental results, the proposed method obtains Precision 5 (P5), P10, Recall, Mean Reciprocal Rate (MRR), and Success Rate (SR@K) of 21%, 14%, 42%, 42%, and 58%, respectively. These results indicate that PRF with a combination of statistical and semantic term extraction for searching Arabic documents can improve the relevant document results.

show abstract

BERT-QE: Contextualized Query Expansion for Document Re-ranking

Cited by 56 publications

References 26 publications

Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval

Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval

CEQE: Contextualized Embeddings for Query Expansion

Pseudo-Relevance Feedback Combining Statistical and Semantic Term Extraction for Searching Arabic Documents

Contact Info

Product

Resources

About