AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts

Shin, Taylor; Razeghi, Yasaman; Logan, Robert L.; Wallace, Eric; Singh, Sameer

doi:10.48550/arxiv.2010.15980

Cited by 116 publications

(158 citation statements)

References 11 publications

Supporting

Mentioning

156

Contrasting

Order By: Relevance

“…Therefore, an approach that does not require tuning the large model is highly desired, which is termed as promptbased learning. Based on the format of prompts, the prompt-based learning can be categorized into two kinds: discrete prompt (Jiang et al, 2020;Yuan et al, 2021;Haviv et al, 2021;Wallace et al, 2019;Shin et al, 2020;Gao et al, 2021;Ben-David et al, 2021;Davison et al, 2019) and continuous prompt (Zhong et al, 2021;Qin and Eisner, 2021;Hambardzumyan et al, 2021;Liu et al, 2021b;. The discrete prompt is usually a sequence of tokens or natural language phrases while the continuous prompt is designed as a sequence of vectors (embedding).…”

Section: Prompts For Pre-trained Modelsmentioning

confidence: 99%

See 1 more Smart Citation

Black-box Prompt Learning for Pre-trained Language Models

Diao¹,

Li²,

Ye³

et al. 2022

Preprint

View full text Add to dashboard Cite

Domain-specific fine-tuning strategies for large pre-trained models received vast attention in recent years. In previously studied settings, the model architectures and parameters are tunable or at least visible, which we refer to as white-box settings. This work considers a new scenario, where we do not have access to a pre-trained model, except for its outputs given inputs, and we call this problem blackbox fine-tuning. To illustrate our approach, we first introduce the BLACK-BOX setting formally on text classification, where the pretrained model is not only frozen but also invisible. We then propose our solution BLACK-BOX prompt, a new technique in the promptlearning family, which can leverage the knowledge learned by pre-trained models from the pre-training corpus. Our experiments demonstrate that the proposed method achieved the state-of-the-art performance on eight datasets. Further analyses on different human-designed objectives, prompt lengths, and intuitive explanations demonstrate the robustness and flexibility of our method.

show abstract

Section: Prompts For Pre-trained Modelsmentioning

confidence: 99%

“…It is well known that prompt-based methods are sensitive to the many aspects of prompts including contexts (Jiang et al, 2020;Shin et al, 2020), orders (Lu et al, 2021) and length , and inappropriate prompts will cause bad performance.…”

Section: Effects Of Prompt Lengthmentioning

confidence: 99%

Black-box Prompt Learning for Pre-trained Language Models

Diao¹,

Li²,

Ye³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Different from the traditional approaches that encode the sentence into a set of vectors and then classify their sentiment through a fully connected layer, the prompt-based method will construct a set of templates, for example: ("I am always happy to see you, the sentence's sentiment is [MASK]"), and then ask the model to predict the token [mask] according to the original training task for the PLM. This approach has gone through various stages, from manual template construction [Jiang et al 2020], to automated search for discrete tokens [Shin et al 2020], to continuous virtual Tokon representations [Lester et al 2021;Li and Liang 2021]. It has achieved a great success in few-shot scenarios.…”

Section: Fine-tuningmentioning

confidence: 99%

A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models

Zhang¹,

Song²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that are more natural and better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the lower level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks which may require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize CTG techniques from the perspective of PLMs. We hope it can help researchers in related fields to quickly track the academic frontier, providing them with a landscape of the area and a roadmap for future research.

show abstract

“…Most works use human-written verbalizers (Schick and Schütze, 2020a) ased towards personal vocabulary and do not have enough coverage. Some other studies (Gao et al, 2020;Shin et al, 2020;Schick et al, 2020) design automatic verbalizer searching methods for better verbalizer choices, however, their methods require adequate training set and validation set for optimization. Moreover, the automatically determined verbalizers are usually synonym of the class name, which differs from our intuition of expanding the verbalizer with a set of diverse and comprehensive label words using external KB.…”

Section: Related Workmentioning

confidence: 99%

Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification

Hu¹,

Ding²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Tuning pre-trained language models (PLMs) with task-specific prompts has been a promising approach for text classification. Particularly, previous studies suggest that prompttuning has remarkable superiority in the lowdata scenario over the generic fine-tuning methods with extra classifiers. The core idea of prompt-tuning is to insert text pieces, i.e., template, to the input and transform a classification problem into a masked language modeling problem, where a crucial step is to construct a projection, i.e., verbalizer, between a label space and a label word space. A verbalizer is usually handcrafted or searched by gradient descent, which may lack coverage and bring considerable bias and high variances to the results. In this work, we focus on incorporating external knowledge into the verbalizer, forming a knowledgeable prompttuning (KPT), to improve and stabilize prompttuning. Specifically, we expand the label word space of the verbalizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the expanded label word space. Extensive experiments on zero and few-shot text classification tasks demonstrate the effectiveness of knowledgeable prompt-tuning.

show abstract

AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts

Cited by 116 publications

References 11 publications

Black-box Prompt Learning for Pre-trained Language Models

Black-box Prompt Learning for Pre-trained Language Models

A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models

Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification

Contact Info

Product

Resources

About