Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm

Abstract

Prevailing methods for mapping large generative language models to supervised tasks may fail to sufficiently probe models' novel capabilities. Using GPT-3 as a case study, we show that 0-shot prompts can significantly outperform few-shot prompts. We suggest that the function of few-shot examples in these cases is better described as locating an already learned task rather than meta-learning. This analysis motivates rethinking the role of prompts in controlling and evaluating powerful language models. In this work, we discuss methods of prompt programming, emphasizing the usefulness of considering prompts through the lens of natural language. We explore techniques for exploiting the capacity of narratives and cultural anchors to encode nuanced intentions and techniques for encouraging deconstruction of a problem into components before producing a verdict. Informed by this more encompassing theory of prompt programming, we also introduce the idea of a metaprompt that seeds the model to generate its own natural language prompts for a range of tasks. Finally, we discuss how these more general methods of interacting with language models can be incorporated into existing and future benchmarks and practical applications.

Surface Form Competition: Why the Highest Probability Answer Isn't Always Right

“…For instance placing "TL;DR" (internet slang for Too Long; Didn't Read) at the end of an article causes the model to generate a summary. Efficiently discovering the right prompt is difficult and has become an active area of research (Reynolds and McDonell, 2021;Shin et al, 2020;Jiang et al, 2020). Brown et al (2020) demonstrated that few-shot learning without fine-tuning is possible with very large language models.…”

What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers

“…Prompt-based approaches involve constructing optimal prompts for language models to best elicit knowledge and maximize prediction performances (Radford et al, 2019;Schick and Schütze, 2020). As the scale of language models grows, the potential of replacing the full finetuning paradigm with the prompt-based approach has been reported (Reynolds and McDonell, 2021;Li and Liang, 2021), as learning via prompts is efficient regarding time and space complexity. However, language models are highly sensitive to the prompt design, motivating methodologies for optimizing prompts.…”

Sparks: Inspiration for Science Writing using Language Models

“…It has been shown that a well-selected prefix, or 'prompt', can dramatically increase the performance of a language model on a specific task [41]. A resulting line of research has been automatically creating either natural language prompts or continuous vector prompts, to perform well on tasks [19,34].…”

“…First we craft a 'prefix' prompt to pre-pend to any prompt used by a writer. Prefix prompts have been shown to greatly improve performance by providing the language model with appropriate context [41]. We found early on in development that simply providing the model with a technical topic was not enough -also providing a context area was necessary for it to appropriately interpret technical terms.…”