DKPLM: Decomposable Knowledge-Enhanced Pre-trained Language Model for Natural Language Understanding

Zhang, Taolin; Wang, Chengyu; Hu, Nan; Qiu, Minghui; Tang, Chengguang; He, Xiaofeng; Huang, Jun

doi:10.1609/aaai.v36i10.21425

Cited by 19 publications

(5 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Knowledge-enhanced Pre-training. Conventional pre-training methods lack factual knowledge (Zhang et al, 2022b;. To deal with this issue, we present KP-PLM with a novel knowledge prompting paradigm for knowledge-enhanced pre-training.…”

Section: Core Capacitiesmentioning

confidence: 99%

HugNLP: A Unified and Comprehensive Library for Natural Language Processing

Wang¹,

Sun²,

Huang³

et al. 2023

Preprint

View full text Add to dashboard Cite

In this paper, we introduce HugNLP, a unified and comprehensive library for natural language processing (NLP) with the prevalent backend of HuggingFace Transformers, which is designed for NLP researchers to easily utilize off-the-shelf algorithms and develop novel methods with user-defined models and tasks in real-world scenarios. HugNLP consists of a hierarchical structure including models, processors and applications that unifies the learning process of pre-trained language models (PLMs) on different NLP tasks. Additionally, we present some featured NLP applications to show the effectiveness of HugNLP, such as knowledgeenhanced PLMs, universal information extraction, low-resource mining, and code understanding and generation, etc. The source code will be released on GitHub (https:// github.com/wjn1996/HugNLP).

show abstract

Section: Core Capacitiesmentioning

confidence: 99%

HugNLP: A Unified and Comprehensive Library for Natural Language Processing

Wang¹,

Sun²,

Huang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Knowledge-based PLM (KEPLM) improved the semantic understanding capability of the model by incorporating the relational triples in the knowledge graph into the language model pre-training process. DKPLM [17] had found a suitable pre-training task for injecting the knowledge graph. This can help the model to improve its ability to learn and represent knowledge without introducing additional parameters and models.…”

Section: Plmmentioning

confidence: 99%

EPTQA:a Chinese medical prompt learning method based on entity pair type question answering

Su,

Yu,

et al. 2023

Preprint

View full text Add to dashboard Cite

As a mainstream method in current natural language processing tasks, the “pre-train, fine-tune” method has achieved quite good results in various scenarios. However, the “pre-train, fine-tune” method performs poorly on few-shot learning tasks. Recently, prompt learning has gained popularity. Prompt learning transforms various types of natural language processing tasks into pre-training tasks and shows good results on few-shot learning tasks. The prompt learning method based on entity pair type question answering proposed in this paper creatively applies the prompt learning method successfully to the Chinese medical relationship extraction task. The proposed model shows good results on both full data and and low resource datasets. Background Chinese medical relation extraction is an important step in building a complete medical knowledge system. Although the “pre-train, fine-tune” paradigm has shown good results in the Chinese medical relationship extraction task, the “pre-train, fine-tune” paradigm has slow model convergence, and the results are not satisfactory in the small-sample relationship extraction task. These problems are related to the scarcity of accurately labelled medical text data and the large differences between upstream and downstream models. Results Given the aforementioned problems, we propose a prompt learning method that is based on entity pair type question answering. To start with, we preprocessed the Chinese medical text dataset by transforming the data into a sentence-level relation extraction form, which is more appropriate for prompt learning. The relationship template is then devised by combining entity types, which effectively address the issue of expressing the Chinese medical relationship in an accurate and brief manner. Following the fine-tuning of the pre-trained model, this method can accurately restore the mask and present very competitive outcomes on the full data and low resource data of numerous Chinese medical datasets. Conclusions The method proposed in this paper is superior to the traditional “pre-train, fine-tune” approach, as it can efficiently extract the connections between Chinese medical entities. This method is particularly effective when dealing with small sample datasets. The efficiency of the model can be further improved by using the relationship filtering method which is based on the relevant rules of the Chinese medical entities.

show abstract

“…There are substantial works to tackle these subtasks using PLMs as the cornerstone and with success [28][29][30][31][32]. Moreover, several works utilize structured knowledge to enhance PLMs for more than just self-supervised training on the large-scale corpus, with significant improvements in mention detection [33] and relation detection [34][35][36][37] tasks. Nevertheless , there is lack of work on making a comprehensive comparison on each subtasks from the perspective of PLMs.…”

Section: Introductionmentioning

confidence: 99%

An Empirical Study of Pre-trained Language Models in Simple Knowledge Graph Question Answering

Hu¹,

Wu²,

Qi³

et al. 2023

Preprint

View full text Add to dashboard Cite

Large-scale pre-trained language models (PLMs) such as BERT have recently achieved great success and become a milestone in natural language processing (NLP). It is now the consensus of the NLP community to adopt PLMs as the backbone for downstream tasks. In recent works on knowledge graph question answering (KGQA), BERT or its variants have become necessary in their KGQA models. However, there is still a lack of comprehensive research and comparison of the performance of different PLMs in KGQA. To this end, we summarize two basic KGQA frameworks based on PLMs without additional neural network modules to compare the performance of nine PLMs in terms of accuracy and efficiency. In addition, we present three benchmarks for larger-scale KGs based on the popular SimpleQuestions benchmark to investigate the scalability of PLMs. We carefully analyze the results of all PLMs-based KGQA basic frameworks on these benchmarks and two other popular datasets, WebQuestionSP and FreebaseQA, and find that knowledge distillation techniques and knowledge enhancement methods in PLMs

show abstract

DKPLM: Decomposable Knowledge-Enhanced Pre-trained Language Model for Natural Language Understanding

Cited by 19 publications

References 35 publications

HugNLP: A Unified and Comprehensive Library for Natural Language Processing

HugNLP: A Unified and Comprehensive Library for Natural Language Processing

EPTQA:a Chinese medical prompt learning method based on entity pair type question answering

An Empirical Study of Pre-trained Language Models in Simple Knowledge Graph Question Answering

Contact Info

Product

Resources

About