Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

Abstract

This paper surveys and organizes research works in a new paradigm in natural language processing, which we dub “prompt-based learning”. Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P (y|x), prompt-based learning is based on language models that model the probability of text directly. To use these models to perform prediction tasks, the original input x is modified using a template into a textual string prompt x′ that has some unfilled slots, and then the language model is used to probabilistically fill the unfilled information to obtain a final string \(\hat{\bm {x}} \) , from which the final output y can be derived. This framework is powerful and attractive for a number of reasons: it allows the language model to be pre-trained on massive amounts of raw text, and by defining a new prompting function the model is able to perform few-shot or even zero-shot learning, adapting to new scenarios with few or no labeled data. In this paper we introduce the basics of this promising paradigm, describe a unified set of mathematical notations that can cover a wide variety of existing work, and organize existing work along several dimensions, e.g. the choice of pre-trained language models, prompts, and tuning strategies. To make the field more accessible to interested beginners, we not only make a systematic review of existing works and a highly structured typology of prompt-based concepts, but also release other resources, e.g., a website NLPedia–Pretrain including constantly-updated survey, and paperlist.

Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies

“…Prompt design. Liu et al (2021) survey methods for designing prompts. OpenAI's best practices 1 for designing prompts include giving clear instructions alongside a few illustrative examples, called a few-shot prompt (Brown et al 2020).…”

Analysis and Application of the Business English Translation Query and Decision Model with Big Data Corpus

“…is a feed-forward neural network that has been widely used in recent years in text classification, computer vision, and other fields [15]. e translation query and decision system constructed in this paper firstly use a convolution network as a feature extractor to realize the feature extraction of input text.…”

Research on Implicit Intent Recognition Method Based on Prompt Learning

“…Prompt learning is the new paradigm of natural language processing, which can prompt the pretrained model to recall the learned knowledge in the pre-training stage by constructing a suitable prompt, so as to unify the downstream tasks with the pre-training tasks [18]. The literature [26] is an earlier work on prompt learning and proposes a PET method which is a semi-supervised training process.…”