Language Models are Few-Shot Learners

Abstract

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions -something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art finetuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

Learning to Compose Diversified Prompts for Image Emotion Classification

“…With the application of large-scale pre-trained language models such as BERT [Devlin et al, 2018] and GPT [Brown et al, 2020], many downstream applications have achieved significant improvements by finetuning on top of the pre-trained models. However, due to the large scale of model parameters, finetuning brings a large computational and storage burden.…”

“…Currently, prompt tuning methods can be roughly divided into two categories: manually crafted and automatically learned. While manually crafting prompts [Brown et al, 2020] [Radford et al, 2021is intuitive, creating and experimenting with these prompts takes time and experience, even experienced prompt designers may fail to manually discover optimal prompts . To automate prompt engineering, ] [Lester et al, 2021] [Zhou et al, 2021 paramerized the prompts by treating prompts as virtual tokens and perform prompting directly in the embedding space.…”

All-optical ultrafast ReLU function for energy-efficient nanophotonic deep learning

“…4(b). The GELU func- tion is used extensively in Transformer networks for natural language processing, which are regularly amongst the largest deep learning models [29]. Thus, our alloptical PPLN nanophotonic waveguide implementation gains greater real-world applicability by being compatible with a wide range of existing deep learning models, especially the largest models where energy efficiency is paramount.…”

Cross-modal Contrastive Distillation for Instructional Activity Anticipation

“…Text knowledge for vision tasks. Recent studies [35], [36] explore using large-scale language models [37] for vision tasks and observe significant improvements with text knowledge. We expect CCD provides a better way of utilizing the text knowledge, thus further improving the vision task performance.…”