Enriching Pre-trained Language Model with Entity Information for Relation Classification

Wu, Shanchan; He, Yifan

doi:10.1145/3357384.3358119

Cited by 339 publications

(225 citation statements)

References 13 publications

Supporting

Mentioning

221

Contrasting

Unclassified

Order By: Relevance

“…Table II shows that our model obtains an F1-score of 90.36%, outperforming the state-of-the-art models substantially. The best results of the CNN-based and RNN-based models range from 84% to 86%, while the recent R-BERT model proposed by Wu and He [24] obtains the best F1score of 89.25%, which has an approximately 4-point gap with previous methods. It is noteworthy that the proposed relation extraction model introducing syntactic indicators has a further performance improvement in this task.…”

Section: Resultsmentioning

confidence: 92%

“…It has been applied to multiple NLP tasks and obtains new startof-the-art results on eleven tasks, such as text classification, sequence labeling, and question answering. In recent research, Wu and He [24] propose an R-BERT model, which employs the pre-trained BERT language model and reaches the top of the leaderboard in relation extraction.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Enhancing Relation Extraction Using Syntactic Indicators and Sentential Contexts

Tao

Luo

Wang

et al. 2019

2019 IEEE 31st International Conference on Tools With Artificial Intelligence (ICTAI)

View full text Add to dashboard Cite

State-of-the-art methods for relation extraction consider the sentential context by modeling the entire sentence. However, syntactic indicators, certain phrases or words like prepositions that are more informative than other words and may be beneficial for identifying semantic relations. Other approaches using fixed text triggers capture such information but ignore the lexical diversity. To leverage both syntactic indicators and sentential contexts, we propose an indicator-aware approach for relation extraction. Firstly, we extract syntactic indicators under the guidance of syntactic knowledge. Then we construct a neural network to incorporate both syntactic indicators and the entire sentences into better relation representations. By this way, the proposed model alleviates the impact of noisy information from entire sentences and breaks the limit of text triggers. Experiments on the SemEval-2010 Task 8 benchmark dataset show that our model significantly outperforms the state-of-the-art methods.

show abstract

Section: Resultsmentioning

confidence: 92%

Section: Related Workmentioning

confidence: 99%

Enhancing Relation Extraction Using Syntactic Indicators and Sentential Contexts

Tao

Luo

Wang

et al. 2019

2019 IEEE 31st International Conference on Tools With Artificial Intelligence (ICTAI)

View full text Add to dashboard Cite

show abstract

“…Relation extraction from text is a popular task; many publications show that neural methods, particularly RNNs and LSTMs, perform substantially better than non-neural ones [8,9,11,16,18]. No such neural methods exist for relation extraction on tables, however.…”

Section: Related Workmentioning

confidence: 99%

“…We chose LSTMs because they have been shown to perform well on NLP tasks, including relation extraction from text [8,11,18], due to their ability of representing sentences based on their salient words. We used the following contextual information: the title of the section containing the table, the first paragraph in that section, the headers and the caption of the table (when present).…”

Section: Neural Networkmentioning

confidence: 99%

Neural Relation Extraction on Wikipedia Tables for Augmenting Knowledge Graphs

Macdonald¹,

Barbosa

2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

Knowledge Graph Augmentation is the task of adding missing facts to an incomplete knowledge graph to improve its effectiveness in applications such as web search and question answering. State-ofthe-art methods rely on information extraction from running text, leaving rich sources of facts such as tables behind. We help close this gap with a neural method that uses contextual information surrounding a table in a Wikipedia article to extract relations between entities appearing in the same row of a table or between the entity of said article and entities appearing in the table. We trained and tested our method on a much larger dataset compared to previous work which we have made public and observed experimentally that our method is very promising for the task.

show abstract

“…For instance, sentence classification tasks with the original BERT model is possible by passing the sentence representation token (denoted [CLS]) through a linear layer. More recent work (specific to the task of relationship extraction) has explored how combining embedded entity information with such sentence representations can lead to significant performance boosts (the RBERT head) 10 . However, evidence has since emerged 11 that at least some of the perceived performance gains of transformer style models is due to so-called 'Clever Hans' type effects, where the model is fine-tuned to learn unintended correlations in datasets rather than a generalised representation of the task.…”

Section: Introductionmentioning

confidence: 99%

Ablations over transformer models for biomedical relationship extraction

et al. 2020

View full text Add to dashboard Cite

Background: Masked language modelling approaches have enjoyed success in improving benchmark performance across many general and biomedical domain natural language processing tasks, including biomedical relationship extraction (RE). However, the recent surge in both the number of novel architectures and the volume of training data they utilise may lead us to question whether domain specific pretrained models are necessary. Additionally, recent work has proposed novel classification heads for RE tasks, further improving performance. Here, we perform ablations over several pretrained models and classification heads to try to untangle the perceived benefits of each. Methods: We use a range of string preprocessing strategies, combined with Bidirectional Encoder Representations from Transformers (BERT), BioBERT and RoBERTa architectures to perform ablations over three RE datasets pertaining to drug-drug and chemical protein interactions, and general domain relationship extraction. We explore the use of the RBERT classification head, compared to a simple linear classification layer across all architectures and datasets. Results: We observe a moderate performance benefit in using the BioBERT pretrained model over the BERT base cased model, although there appears to be little difference when comparing BioBERT to RoBERTa large. In addition, we observe a substantial benefit of using the RBERT head on the general domain RE dataset, but this is not consistently reflected in the biomedical RE datasets. Finally, we discover that randomising the token order of training data does not result in catastrophic performance degradation in our selected tasks. Conclusions: We find a recent general domain pretrained model performs approximately the same as a biomedical specific one, suggesting that domain specific models may be of limited use given the tendency of recent model pretraining regimes to incorporate ever broader sets of data. In addition, we suggest that care must be taken in RE model training, to prevent fitting to non-syntactic features of datasets.

show abstract

Enriching Pre-trained Language Model with Entity Information for Relation Classification

Cited by 339 publications

References 13 publications

Enhancing Relation Extraction Using Syntactic Indicators and Sentential Contexts

Enhancing Relation Extraction Using Syntactic Indicators and Sentential Contexts

Neural Relation Extraction on Wikipedia Tables for Augmenting Knowledge Graphs

Ablations over transformer models for biomedical relationship extraction

Contact Info

Product

Resources

About