LightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Prompting

Chen, Xiang; Li, Lei; Deng, Shumin; Tan, Chuanqi; Chen, Xu; Huang, Fei; Si, Luo; Chen, Huajun; Zhang, Ningyu

doi:10.48550/arxiv.2109.00720

Cited by 5 publications

(5 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prompting's advantage and effectiveness in a wide range of NLP applications has been verified in recent literature, including text classification (Hu et al, 2021;Min et al, 2021;, entity typing , few-shot learning Xu et al, 2021a;Zhao et al, 2021;, relation extraction Han et al, 2021b;Sainz et al, 2021), knowledge probing (Zhong et al, 2021), named entity recognition (Chen et al, 2021b), machine translation (Tan et al, 2021;Wang et al, 2021b) and dialogue system (Wang et al, 2021a).…”

Section: Related Workmentioning

confidence: 95%

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

Liu¹,

Ji²,

Fu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Prompt tuning, which only tunes continuous prompts with a frozen language model, substantially reduces per-task storage and memory usage at training. However, in the context of NLU, prior work reveals that prompt tuning does not perform well for normal-sized pretrained models. We also find that existing methods of prompt tuning cannot handle hard sequence tagging tasks, indicating a lack of universality. We present a novel empirical finding that properly optimized prompt tuning can be universally effective across a wide range of model scales and NLU tasks. It matches the performance of finetuning while having only 0.1%-3% tuned parameters. Our method P-Tuning v2 is not a new method, but a version of prefix-tuning (Li and Liang, 2021) optimized and adapted for NLU. Given the universality and simplicity of P-Tuning v2, we believe it can serve as an alternative to finetuning and a strong baseline for future research. 1

show abstract

Section: Related Workmentioning

confidence: 95%

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

Liu¹,

Ji²,

Fu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Whereas, additive attention scores between word queries and picture features were used to weigh and calculate the visual attention features using VGG-19 visual features (Simonyan and Zisserman, 2014). Chen S. et al (2020) andChen X. et al (2021) extracted visual information into subtitles and proposed a softer method of image-text combination that improves the fusion of different modal features.…”

Section: Named Entity Recognitionmentioning

confidence: 99%

MLNet: a multi-level multimodal named entity recognition architecture

Zhai

Hou

et al. 2023

Front. Neurorobot.

View full text Add to dashboard Cite

In the field of human–computer interaction, accurate identification of talking objects can help robots to accomplish subsequent tasks such as decision-making or recommendation; therefore, object determination is of great interest as a pre-requisite task. Whether it is named entity recognition (NER) in natural language processing (NLP) work or object detection (OD) task in the computer vision (CV) field, the essence is to achieve object recognition. Currently, multimodal approaches are widely used in basic image recognition and natural language processing tasks. This multimodal architecture can perform entity recognition tasks more accurately, but when faced with short texts and images containing more noise, we find that there is still room for optimization in the image-text-based multimodal named entity recognition (MNER) architecture. In this study, we propose a new multi-level multimodal named entity recognition architecture, which is a network capable of extracting useful visual information for boosting semantic understanding and subsequently improving entity identification efficacy. Specifically, we first performed image and text encoding separately and then built a symmetric neural network architecture based on Transformer for multimodal feature fusion. We utilized a gating mechanism to filter visual information that is significantly related to the textual content, in order to enhance text understanding and achieve semantic disambiguation. Furthermore, we incorporated character-level vector encoding to reduce text noise. Finally, we employed Conditional Random Fields for label classification task. Experiments on the Twitter dataset show that our model works to increase the accuracy of the MNER task.

show abstract

“…Prompt-tuning for pre-trained language models is a rapidly emerging field in natural language processing [40,46,71] and have attracted lots of attention. Originally from GPT-3, prompt-tuning has been applied to various of tasks including relation extraction [20], event extraction [21,59], named entity recognition [5,7], entity typing [13], and so on. To facilitate the labor-intensive prompt engineering, [43] propose AU-TOPROMPT, which can search prompts based on a gradient method to select label words and templates.…”

Section: Prompt-tuningmentioning

confidence: 99%

Ontology-enhanced Prompt-tuning for Few-shot Learning

Ye,

Zhang,

Deng

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Few-shot Learning (FSL) is aimed to make predictions based on a limited number of samples. Structured data such as knowledge graphs and ontology libraries has been leveraged to benefit the few-shot setting in various tasks. However, the priors adopted by the existing methods suffer from challenging knowledge missing, knowledge noise, and knowledge heterogeneity, which hinder the performance for few-shot learning. In this study, we explore knowledge injection for FSL with pre-trained language models and propose ontology-enhanced prompt-tuning (OntoPrompt). Specifically, we develop the ontology transformation based on the external knowledge graph to address the knowledge missing issue, which fulfills and converts structure knowledge to text. We further introduce span-sensitive knowledge injection via a visible matrix to select informative knowledge to handle the knowledge noise issue. To bridge the gap between knowledge and text, we propose a collective training algorithm to optimize representations jointly. We evaluate our proposed OntoPrompt in three tasks, including relation extraction, event extraction, and knowledge graph completion, with eight datasets. Experimental results demonstrate that our approach can obtain better few-shot performance than baselines.

show abstract

LightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Prompting

Cited by 5 publications

References 38 publications

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

MLNet: a multi-level multimodal named entity recognition architecture

Ontology-enhanced Prompt-tuning for Few-shot Learning

Contact Info

Product

Resources

About