Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification

Hu, Shengding; Ding, Ning; Wang, Huadong; Liu, Zhiyuan; Wang, Jingang; Li, Juanzi; Wu, Wei; Sun, Maosong

doi:10.48550/arxiv.2108.02035

Cited by 36 publications

(42 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prompting's advantage and effectiveness in a wide range of NLP applications has been verified in recent literature, including text classification (Hu et al, 2021;Min et al, 2021;, entity typing , few-shot learning Xu et al, 2021a;Zhao et al, 2021;, relation extraction Han et al, 2021b;Sainz et al, 2021), knowledge probing (Zhong et al, 2021), named entity recognition (Chen et al, 2021b), machine translation (Tan et al, 2021;Wang et al, 2021b) and dialogue system (Wang et al, 2021a).…”

Section: Related Workmentioning

confidence: 95%

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

Liu¹,

Ji²,

Fu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Prompt tuning, which only tunes continuous prompts with a frozen language model, substantially reduces per-task storage and memory usage at training. However, in the context of NLU, prior work reveals that prompt tuning does not perform well for normal-sized pretrained models. We also find that existing methods of prompt tuning cannot handle hard sequence tagging tasks, indicating a lack of universality. We present a novel empirical finding that properly optimized prompt tuning can be universally effective across a wide range of model scales and NLU tasks. It matches the performance of finetuning while having only 0.1%-3% tuned parameters. Our method P-Tuning v2 is not a new method, but a version of prefix-tuning (Li and Liang, 2021) optimized and adapted for NLU. Given the universality and simplicity of P-Tuning v2, we believe it can serve as an alternative to finetuning and a strong baseline for future research. 1

show abstract

Section: Related Workmentioning

confidence: 95%

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

Liu¹,

Ji²,

Fu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…[20] propose an approach called PTR, which leverages logic rules to construct prompts with sub-prompts for many-class text classification. [22] propose an approach to incorporate external knowledge graph into the verbalizer with calibration. [6] propose a knowledge-aware prompt-tuning approach that injects knowledge into prompt template design and answer construction.…”

Section: Prompt-tuningmentioning

confidence: 99%

Ontology-enhanced Prompt-tuning for Few-shot Learning

Ye,

Zhang,

Deng

et al. 2022

Preprint

View full text Add to dashboard Cite

Few-shot Learning (FSL) is aimed to make predictions based on a limited number of samples. Structured data such as knowledge graphs and ontology libraries has been leveraged to benefit the few-shot setting in various tasks. However, the priors adopted by the existing methods suffer from challenging knowledge missing, knowledge noise, and knowledge heterogeneity, which hinder the performance for few-shot learning. In this study, we explore knowledge injection for FSL with pre-trained language models and propose ontology-enhanced prompt-tuning (OntoPrompt). Specifically, we develop the ontology transformation based on the external knowledge graph to address the knowledge missing issue, which fulfills and converts structure knowledge to text. We further introduce span-sensitive knowledge injection via a visible matrix to select informative knowledge to handle the knowledge noise issue. To bridge the gap between knowledge and text, we propose a collective training algorithm to optimize representations jointly. We evaluate our proposed OntoPrompt in three tasks, including relation extraction, event extraction, and knowledge graph completion, with eight datasets. Experimental results demonstrate that our approach can obtain better few-shot performance than baselines.

show abstract

“…Although the expected answer may be in the form of tokens, spans, or sentences in token-level promptlearning, the predicted answer is always generated token by token. Tokens are usually mapped to the whole vocabulary or a set of candidate words (Petroni et al 2019;Cui et al 2021;Han et al 2021;Adolphs, Dhuliawala, and Hofmann 2021;Hu et al 2021). Take PET model (Schick and Schütze 2021b,a) as an example, the sentiment classification input/label pair is reformulated to "x: [CLS] The Italian team won the European Cup.…”

Section: Token-level and Sentence-levelmentioning

confidence: 99%

NSP-BERT: A Prompt-based Zero-Shot Learner Through an Original Pre-training Task--Next Sentence Prediction

Sun¹,

Zheng²,

Hao³

et al. 2021

Preprint

View full text Add to dashboard Cite

Using prompts to utilize language models to perform various downstream tasks, also known as prompt-based learning or prompt-learning, has lately gained significant success in comparison to the pre-train and fine-tune paradigm. Nonetheless, virtually all prompt-based methods are tokenlevel, meaning they all utilize GPT's left-to-right language model or BERT's masked language model to perform clozestyle tasks. In this paper, we attempt to accomplish several NLP tasks in the zero-shot scenario using a BERT original pre-training task abandoned by RoBERTa and other models-Next Sentence Prediction (NSP). Unlike token-level techniques, our sentence-level prompt-based method NSP-BERT does not need to fix the length of the prompt or the position to be predicted, allowing it to handle tasks such as entity linking with ease. Based on the characteristics of NSP-BERT, we offer several quick building templates for various downstream tasks. We suggest a two-stage prompt method for word sense disambiguation tasks in particular. Our strategies for mapping the labels significantly enhance the model's performance on sentence pair tasks. On the FewCLUE benchmark, our NSP-BERT outperforms other zero-shot methods on most of these tasks and comes close to the few-shot methods. 1

show abstract

Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification

Cited by 36 publications

References 19 publications

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

Ontology-enhanced Prompt-tuning for Few-shot Learning

NSP-BERT: A Prompt-based Zero-Shot Learner Through an Original Pre-training Task--Next Sentence Prediction

Contact Info

Product

Resources

About