MarkedBERT: Integrating Traditional IR Cues in Pre-trained Language Models for Passage Retrieval

Boualili, Lila; Moreno, José G.

doi:10.1145/3397271.3401194

Cited by 28 publications

(26 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For Siamese BERT, we follow the structure described in the paper (Reimers and Gurevych, 2019) for this experiment. Besides, we have tried to implement more recent matching models (Liu et al, 2018b;Hofstätter, 2020) and BERT-based variant models (Boualili et al, 2020; Rudra and Anand, 2020), but we do not obtain the expected excellent results. For fair comparison, these methods are not included in this paper.…”

Section: A3 Comparison Methodsmentioning

confidence: 92%

Leveraging Capsule Routing to Associate Knowledge with Medical Literature Hierarchically

Liu

Chen

Chen³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Integrating knowledge into text is a promising way to enrich text representation, especially in the medical field. However, undifferentiated knowledge not only confuses the text representation but also imports unexpected noises. In this paper, to alleviate this problem, we propose leveraging capsule routing to associate knowledge with medical literature hierarchically (called HiCapsRKL). Firstly, HiCapsRKL extracts two empirically designed text fragments from medical literature and encodes them into fragment representations respectively. Secondly, the capsule routing algorithm is applied to two fragment representations. Through the capsule computing and dynamic routing, each representation is processed into a new representation (denoted as caps-representation), and we integrate the caps-representations as information gain to associate knowledge with medical literature hierarchically. Finally, HiCapsRKL are validated on relevance prediction and medical literature retrieval test sets. The experimental results and analyses show that HiCapsRKL can more accurately associate knowledge with medical literature than the mainstream methods. In summary, HiCapsRKL can efficiently help selecting the most relevant knowledge to the medical literature, which may be an alternative attempt to improve knowledge-based text representation. Source code is released on GitHub 1 .

show abstract

Section: A3 Comparison Methodsmentioning

confidence: 92%

Leveraging Capsule Routing to Associate Knowledge with Medical Literature Hierarchically

Liu

Chen

Chen³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…In this section, we introduce recent works designing PTMs tailored for IR (Lee et al, 2019b;Chang et al, 2019;Ma et al, 2021b;Ma et al, 2021c;Boualili et al, 2020;Ma et al, 2021d;Zou et al, 2021;Liu et al, 2021d). General pre-trained models like BERT have achieved great success when applied to IR tasks on both the firststage retrieval and the re-ranking stage.…”

Section: Keyphrase Extractionmentioning

confidence: 99%

Pre-training Methods in Information Retrieval

Fan¹,

Xie²,

Cai³

et al. 2021

Preprint

View full text Add to dashboard Cite

The core of information retrieval (IR) is to identify relevant information from large-scale resources and return it as a ranked list to respond to user's information need. Recently, the resurgence of deep learning has greatly advanced this field and leads to a hot topic named NeuIR (i.e., neural information retrieval), especially the paradigm of pre-training methods (PTMs). Owing to sophisticated pre-training objectives and huge model size, pre-trained models can learn universal language representations from massive textual data, which are beneficial to the ranking task of IR. Since there have been a large number of works dedicating to the

show abstract

“…As explained earlier, exact matching is an important matching signal in traditional IR models, and relevance matching-based neural ranking models incorporate the exact matching signal to improve retrieval results. In order to directly incorporate exact matching signal in the sentence pair classification setting of BERT for document retrieval, Boualili et al (2020) proposed to mark the start and end of exact matching query tokens in a document with special markers.…”

Section: Deep Contextualized Language Model-based Representationsmentioning

confidence: 99%

“…Second, some BERTbased models, such as MacAvaney et al (2019), are combined with existing relevancebased ranking models that are asymmetric, and others, such as Nogueira et al (2019), include a component for pairwise comparison of documents, so that the joint model is asymmetric in both cases. Third, Boualili et al (2020) include the exact matching of query tokens into the ranking model which leads to an overall asymmetric architecture. In the case of short documents where BERT can accept the full document and only the BERTbased model is used for ranking, the ranking model is symmetric.…”

Section: Exact Matchingmentioning

confidence: 99%

Neural ranking models for document retrieval

et al. 2021

View full text Add to dashboard Cite

Ranking models are the main components of information retrieval systems. Several approaches to ranking are based on traditional machine learning algorithms using a set of hand-crafted features. Recently, researchers have leveraged deep learning models in information retrieval. These models are trained end-to-end to extract features from the raw data for ranking tasks, so that they overcome the limitations of hand-crafted features. A variety of deep learning models have been proposed, and each model presents a set of neural network components to extract features that are used for ranking. In this paper, we compare the proposed models in the literature along different dimensions in order to understand the major contributions and limitations of each model. In our discussion of the literature, we analyze the promising neural components, and propose future research directions. We also show the analogy between document retrieval and other retrieval tasks where the items to be ranked are structured documents, answers, images and videos.

show abstract

MarkedBERT: Integrating Traditional IR Cues in Pre-trained Language Models for Passage Retrieval

Cited by 28 publications

References 6 publications

Leveraging Capsule Routing to Associate Knowledge with Medical Literature Hierarchically

Leveraging Capsule Routing to Associate Knowledge with Medical Literature Hierarchically

Pre-training Methods in Information Retrieval

Neural ranking models for document retrieval

Contact Info

Product

Resources

About