Commonsense Knowledge-Aware Prompt Tuning for Few-Shot NOTA Relation Classification

Abstract

Compared with the traditional few-shot task, the few-shot none-of-the-above (NOTA) relation classification focuses on the realistic scenario of few-shot learning, in which a test instance might not belong to any of the target categories. This undoubtedly increases the task’s difficulty because given only a few support samples, this cannot represent the distribution of NOTA categories in space. The model needs to make full use of the syntactic information and word meaning information learned in the pre-training stage to distinguish the NOTA category and the support sample category in the embedding space. However, previous fine-tuning methods mainly focus on optimizing the extra classifiers (on top of pre-trained language models (PLMs)) and neglect the connection between pre-training objectives and downstream tasks. In this paper, we propose the commonsense knowledge-aware prompt tuning (CKPT) method for a few-shot NOTA relation classification task. First, a simple and effective prompt-learning method is developed by constructing relation-oriented templates, which can further stimulate the rich knowledge distributed in PLMs to better serve downstream tasks. Second, external knowledge is incorporated into the model by a label-extension operation, which forms knowledgeable prompt tuning to improve and stabilize prompt tuning. Third, to distinguish the NOTA pairs and positive pairs in embedding space more accurately, a learned scoring strategy is proposed, which introduces a learned threshold classification function and improves the loss function by adding a new term focused on NOTA identification. Experiments on two widely used benchmarks (FewRel 2.0 and Few-shot TACRED) show that our method is a simple and effective framework, and a new state of the art is established in the few-shot classification field.

PromptORE - A Novel Approach Towards Fully Unsupervised Relation Extraction

“…In addition to "real word" tokens, BERT uses some special tokens: Prompt-Tuning The idea behind prompt-tuning is to benefit from BERT's ability to predict masked tokens ([MASK] tokens). It is already used in few-shot RE by [15,5,35,56]. It can be summarized as follows:…”

“…Design a prompt P, which is a sequence of tokens that includes one [MASK] token. For instance, Lv et al [35] use the template…”

“…[SEP]" P ′ 2 is the same as [35]. However, we cannot choose the optimal prompt from the previous ones nor use automatic methods to generate prompts (such as [25]) as they require access to labeled data.…”

“…Finally, we notice that there is no consensus on the best prompt from the four proposed ones: P ′ 3 is the best for FewRel, P ′ 2 for FewRel NYT and P for FewRel PubMed. This highlights the importance to select and fine-tune prompts to maximize BERT's performances, which is indeed a major research area for prompt-based methods [35,49,20,25]. Under our fully unsupervised setting's goal, it is unfeasible to fine-tune the prompt due to the lack of labeled data, therefore we decide to keep our original P for the sake of fair results.…”

“…Prompt-tuning replaces the usual training by designing a prompt (i.e., a text that is inputted to BERT), able to elicit as much information as possible from the Pretrained Language Model. Prompt-tuning is already used in few-shot RE [35,5,15,56]. We propose to go further and adapt this paradigm to work in a fully unsupervised way.…”