Large Language Models are Zero-Shot Reasoners

Abstract

Pretrained large language models (LLMs) are widely used in many sub-fields of natural language processing (NLP) and generally known as excellent few-shot learners with task-specific exemplars. Notably, chain of thought (CoT) prompting, a recent technique for eliciting complex multi-step reasoning through step-bystep answer examples, achieved the state-of-the-art performances in arithmetics and symbolic reasoning, difficult system-2 tasks that do not follow the standard scaling laws for LLMs. While these successes are often attributed to LLMs' ability for few-shot learning, we show that LLMs are decent zero-shot reasoners by simply adding "Let's think step by step" before each answer. Experimental results demonstrate that our Zero-shot-CoT, using the same single prompt template, significantly outperforms zero-shot LLM performances on diverse benchmark reasoning tasks including arithmetics (MultiArith, GSM8K, AQUA-RAT, SVAMP), symbolic reasoning (Last Letter, Coin Flip), and other logical reasoning tasks (Date Understanding, Tracking Shuffled Objects), without any hand-crafted fewshot examples, e.g. increasing the accuracy on MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% with an off-the-shelf 175B parameter model. The versatility of this single prompt across very diverse reasoning tasks hints at untapped and understudied fundamental zero-shot capabilities of LLMs, suggesting high-level, multi-task broad cognitive capabilities may be extracted through simple prompting. We hope our work not only serves as the minimal strongest zeroshot baseline for the challenging reasoning benchmarks, but also highlights the importance of carefully exploring and analyzing the enormous zero-shot knowledge hidden inside LLMs before crafting finetuning datasets or few-shot exemplars.Preprint. Under review.

Automatic Generation of Programming Exercises and Code Explanations with Large Language Models

“…In summary, our results support earlier findings that large language models are zero-shot Kojima et al (2022) and few-shot learners Brown et al (2020), meaning that they perform well in tasks even when not given any, or given just a few, task-related examples as input. Our work suggests that modern machine learning models such as OpenAI Codex provide many opportunities for programming course designers, although potential challenges outlined in prior work Finnie-Ansley et al (2022) should not be ignored.…”

“…When experimenting with many other kinds of priming statements for generating the explanations, we found that Codex very rarely provided high-level descriptions. This supports the findings of Kojima et al who found that large language models seem to perform better in reasoning tasks when priming them to "think step by step" Kojima et al (2022). Even a very explicit prompt, such as "A high-level description of the above program:", would still usually result in a line-by-line explanation being produced.…”

Using cognitive psychology to understand GPT-3

“…We are not the first to probe large-scale machine learning models' abilities. Indeed, recently there has been a push towards creating large benchmarks to assess the capability of foundation models [48][49][50] . Large language models have also been studied using other methods from cognitive psychology, such as property induction 51 , thinking-out-loud protocols 52 , or learning causal over-hypotheses 53 , where researchers have come to similar conclusions.…”

Large Language Models Still Can't Plan (A Benchmark for LLMs on Planning and Reasoning about Change)

“…This sparked interest in evaluating the large language models on various reasoning tasks including common-sense reasoning [26,22,7], logical reasoning [24], and even ethical reasoning [12]. The macro-tenor of the drumbeat of these works has been suggesting that LLM's are indeed capable of doing many kinds of reasoning [14,30,4] In this paper, we take a look at the ability of large language models to do reasoning about actions and change which involve common-sense planning tasks. We develop a suit of benchmarks 1 , based on the kinds of domains employed in the International Planning Competition, to test these capabilities.…”