A Novel Document-Level Relation Extraction Method Based on BERT and Entity Information

Abstract

Document-level relation extraction aims to extract the relationship among the entities in a paragraph of text. Compared with sentence-level, the text in document-level relation extraction is much longer and contains many more entities. It makes the document-level relation extraction a harder task. The number and complexity of entities make it necessary to provide enough information about the entities for the models in document-level relation extraction. To solve this problem, we put forward a document-level entity mask method with type information (DEMMT), which masks each mention of the entities by special tokens. By using this entity mask method, the model can accurately obtain every mention and type of the entities. Based on DEMMT, we propose a BERT-based one-pass model, through which we can predict the relationships among the entities by processing the text once. We test the proposed model on the DocRED dataset, which is a large scale open-domain document-level relation extraction dataset. The results on the manually annotated part of DocRED show that our approach obtains 6% F1 improvement compared with the state-of-the-art models that do not use pre-trained models and has 2% F1 improvement than BERT which does not use the DEMMT. On the distant supervision generated part of DocRED, the improvement of F1 is 2% compared with no pre-trained models, and 5% compared with pure BERT.

Efficient Automated Processing of the Unstructured Documents Using Artificial Intelligence: A Systematic Literature Review and Future Directions

“…CNN YOLO is also used for object recognition and feature extraction in complex documents such as comics [111]. • BERT: Few studies [83], [13] reported the use of BERT for extracting features from the data. It is suitable for various types of language processing related task in NLP.…”

“…Fine-tuning BERT for NER performance improvement in financial and biomedical documents utilizing the combination of BERT and word embedding is discussed in the study [83]. Utilizing and fine-tuning the BERT model for achieving document-level relation extraction using DocRED-a large scale open-domain document-level relation extraction dataset shows improvement in F1 measure in [13].…”

Multi-Layout Unstructured Invoice Documents Dataset: A Dataset for Template-Free Invoice Processing and Its Evaluation Using AI Approaches

“…BERT can easily deal with ambiguous languages. The study [18] proposed BERT for extracting relationships amongst entities at the document level from the DocRED dataset. Later, the BERT is fine-tuned with different parameters to improve the performance of the entity relation extraction task by 5% (F1 Score -47%) than BERT without fine-tuning ( F1 Score -42 %).…”

Improving Distantly-Supervised Relation Extraction Through BERT-Based Label and Instance Embeddings

“…The latest development of pre-trained LMs relying on transformer architecture [19] has shown to capture semantic and syntactic features better [13], with [31] proving that pretrained LMs significantly improve the performance in text classification tasks, prevent overfitting, and increase sample efficiency. Moreover, works [32], [33] that fine-tune the pretrained LM models (most of them BERT [18]) have shown that simple NNs built on top of pretrained transformer-based models improve performance. Meanwhile, DISTRE model [11] extended GPT [13] to the DS setting by incorporating a multi-instance training mechanism, proving that pre-trained LMs provide a stronger signal for DS than specific linguistic and side-information features [8].…”