Chain of Thought Prompting Elicits Reasoning in Large Language Models

Lee, Jason; Wang, Xuezhi; Schuurmans, Dale; Bosma, Maarten; Chi, Ed; Le, Quoc V.; Zhou, Denny

doi:10.48550/arxiv.2201.11903

Cited by 362 publications

(532 citation statements)

References 0 publications

Supporting

Mentioning

522

Contrasting

Unclassified

Order By: Relevance

“…We evaluate self-consistency on a range of arithmetic reasoning and commonsense reasoning tasks, and find that it improves the reasoning ability of language models by a striking margin. Compared to generating a single chain of thought via greedy decoding (Wei et al, 2022), self-consistency contributes additional absolute improvements of +10.6% on the recent grade-school-math dataset (GSM8K; Cobbe et al, 2021), +14.4% on a recently-compiled challenge dataset over math word problems (SVAMP; Patel et al, 2021), and +23.9% on MultiArith (Roy & Roth, 2015). For commonsense reasoning, we also observe significant gains in CommonsenseQA (Talmor et al, 2019) (+5%), and the AI2 Reasoning Challenge (ARC) dataset (Clark et al, 2018), with +4% and +4.7% absolute accuracy improvement in the easy and challenge sets, respectively.…”

Section: Majority Votementioning

confidence: 99%

“…We leverage this intuition by proposing the following self-consistency method. First, a language model is prompted with a set of manually written chain of thought exemplars (Wei et al, 2022). Next, we sample a set of candidate outputs from the language model's decoder (Ackley et al, 1985;Ficler & Goldberg, 2017;Fan et al, 2018;Holtzman et al, 2018;Radford et al, 2019;Holtzman et al, 2020), which produces diversity in the set of generated reasoning paths.…”

Section: Self-consistency Over Diverse Reasoning Pathsmentioning

confidence: 99%

“…Although language models have demonstrated remarkable success across a range of NLP tasks, their ability to demonstrate reasoning is often seen as a limitation, which cannot be overcome solely by increasing model scale (Rae et al, 2021;BIG-bench collaboration, 2021, inter alia). In response, Wei et al (2022) have proposed chain of thought prompting, which prompts language models to generate a series of short sentences that mimic the reasoning process a person might employ. For example, given the question "Shawn has five toys.…”

Section: Introductionmentioning

confidence: 99%

“…The final answer is 5+4=9.". Chain of thought prompting has been shown to significantly improve language model performance in a variety of multi-step reasoning tasks (Wei et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Self-Consistency Improves Chain of Thought Reasoning in Language Models

Wang¹,

Lee²,

Schuurmans³

et al. 2022

Preprint

Self Cite

132

113

View full text Add to dashboard Cite

We explore a simple ensemble strategy, self-consistency, that significantly improves the reasoning accuracy of large language models. The idea is to sample a diverse set of outputs from a language model and return the most consistent answer in the set. Such ensembling method improves reasoning accuracy when combined with chain of thought prompting. For arithmetic and commonsense reasoning benchmarks we find that self-consistency yields significant accuracy improvements in a variety of datasets, such as GSM8K (+10%), SVAMP (+14%), MultiArith (+24%), CommonsenseQA (+5%) and ARC (easy +4%, challenge +5%).

show abstract

Section: Majority Votementioning

confidence: 99%

Section: Self-consistency Over Diverse Reasoning Pathsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…The final answer is 5+4=9.". Chain of thought prompting has been shown to significantly improve language model performance in a variety of multi-step reasoning tasks (Wei et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Self-Consistency Improves Chain of Thought Reasoning in Language Models

Wang¹,

Lee²,

Schuurmans³

et al. 2022

Preprint

Self Cite

132

113

View full text Add to dashboard Cite

show abstract

“…The design of prompts can have huge impact on PROMPTING, as pointed by many previous works (Mishra et al, 2021a;Wei et al, 2022). In this section, we investigate how prompt design instruct text generation and affects ZEROGEN's performance.…”

Section: Prompt Engineering In Zerogenmentioning

confidence: 96%

ZeroGen: Efficient Zero-shot Learning via Dataset Generation

Ye¹,

Gao²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

There is a growing interest in dataset generation recently due to the superior generative capacity of large pre-trained language models (PLMs). In this paper, we study a flexible and efficient zero-short learning method, ZE-ROGEN. Given a zero-shot task, we first generate a dataset from scratch using PLMs in an unsupervised manner. Then, we train a tiny task model (e.g., LSTM) under the supervision of the synthesized dataset. This approach allows highly efficient inference as the final task model only has orders of magnitude fewer parameters comparing to PLMs (e.g., GPT2-XL). Apart from being annotationfree and efficient, we argue that ZEROGEN can also provide useful insights from the perspective of data-free model-agnostic knowledge distillation, and unreferenced text generation evaluation. Experiments and analysis on different NLP tasks, namely, text classification, question answering, and natural language inference), show the effectiveness of ZEROGEN.

show abstract

“Tipping the Balance”: Human Intervention in Large Language Model Multi‐Agent Debate

Triem,

Ding

2024

Proceedings of the Association for Information Science and Tech

View full text Add to dashboard Cite

Methods for eliciting reasoning from large language models (LLMs) are shifting from filtering natural language “prompts” through contextualized “personas,” towards structuring conversations between LLM instances, or “agents.” This work expands upon LLM multiagent debate by inserting human opinion into the loop of generated conversation. To simulate complex reasoning, LLM instances were given United States district court decisions and asked to debate whether to “affirm” or “not affirm” the decision. Agents were examined in three phases: “synthetic debate,” where one LLM instance simulated a three‐agent discussion; “multiagent debate,” where three LLM instances discussed among themselves; and “human‐AI debate,” where multiagent debate was interrupted by human opinion. During each phase, a nine‐step debate was simulated one‐hundred times, yielding 2,700 total debate steps. Resulting conversations generated by synthetic debate followed a pre‐set cadence, proving them ineffective at simulating individual agents and confirming that mechanism engineering is critical for multiagent debate. Furthermore, the reasoning process backing multiagent decision‐making was strikingly similar to human decision‐making. Finally, it is discovered that while LLMs do weigh human input more heavily than AI opinion, it is only by a small threshold. Ultimately, this work asserts that careful, human‐in‐the‐loop framework is critical for designing value‐aware, agentic AI agents.

show abstract

Chain of Thought Prompting Elicits Reasoning in Large Language Models

Cited by 362 publications

References 0 publications

Self-Consistency Improves Chain of Thought Reasoning in Language Models

Self-Consistency Improves Chain of Thought Reasoning in Language Models

ZeroGen: Efficient Zero-shot Learning via Dataset Generation

“Tipping the Balance”: Human Intervention in Large Language Model Multi‐Agent Debate

Contact Info

Product

Resources

About