Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations

Jung, Jaehun; Qin, Lianhui; Welleck, Sean; Brahman, Faeze; Bhagavatula, Chandra; Bras, Ronan Le; Choi, Yejin

doi:10.48550/arxiv.2205.11822

Cited by 12 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Ozturkler et al (2022) aggregate language model probabilities using mathematical combinators like sum and product. • Jung et al (2022) recursively generate a tree of explanations for a statement, then determine the truth of the statement by treating the inference as a satisfiability problem over these explanations and their logical relations. • Various works, including Fu et al (2021), Dua et al (2022), Guo et al (2022), and, explore decomposition of questions into subquestions, often under the name multi-hop question-answering.…”

Section: Task Decompositionsmentioning

confidence: 99%

Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes

Reppert¹,

Rachbach²,

George³

et al. 2023

Preprint

View full text Add to dashboard Cite

Language models (LMs) can perform complex reasoning either end-to-end, with hidden latent state, or compositionally, with transparent intermediate state. Composition offers benefits for interpretability and safety, but may need workflow support and infrastructure to remain competitive. We describe iterated decomposition, a human-in-the-loop workflow for developing and refining compositional LM programs. We improve the performance of compositions by zooming in on failing components and refining them through decomposition, additional context, chain of thought, etc. To support this workflow, we develop ICE, an open-source tool for visualizing the execution traces of LM programs. We apply iterated decomposition to three real-world tasks and improve the accuracy of LM programs over less compositional baselines on held-out test sets: describing the placebo used in a randomized controlled trial (25% → 65%), evaluating participant adherence to a medical intervention (53% → 70%), and answering NLP questions on the QASPER dataset (38% → 69%). These applications serve as case studies for a workflow that, if automated, could keep ML systems interpretable and safe even as they scale to increasingly complex tasks.

show abstract

Section: Task Decompositionsmentioning

confidence: 99%

Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes

Reppert¹,

Rachbach²,

George³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…AI models are uniquely developed and trained on distinct collections of training data, which can influence the responses retrieved by a user. There are many different approaches to communicating with a text-based AI interface, such as breaking down complex questions or problems into constituent components, refining prompts based on the model's feedback, providing cues such as a general structure or keywords for the model to include in its response and maieutic prompting that requires the model to explain its responses to ensure logical consistency (Jung et al, 2022).…”

Section: Prompt Engineeringmentioning

confidence: 99%

The prompt engineering librarian

Lund

2023

LHTN

View full text Add to dashboard Cite

Purpose In terms of training the public in prompt engineering skills, no single discipline or profession currently takes the lead, presenting an opportunity for professions like librarianship to step into this role. Librarians are already well-equipped to educate the public in a wide range of literacy skills and tasks, so prompt engineering may be a natural progression. The purpose of this paper is to examine the potential role of prompt engineering for library professionals. Design/methodology/approach Prompt engineering is the process of optimizing the text that is provided to an artificial intelligence (A)I model to ensure proper interpretation and the generation of relevant, detailed results. The field of prompt engineering is relatively young, evolving alongside the growth of large language models like ChatGPT and BARD. This conceptual paper will explore prompt engineering as a possible domain of expertise for librarians. Findings This paper delves into the world of prompt engineering, its alignment with the existing roles and expertise of librarians, and the potential emergence of a new role known as the “prompt engineering librarian,” akin to the well-established “information literacy librarian” role that has gained prominence in recent decades. Originality/value The significance of this work lies in exploring the synergy between prompt engineering and the traditional roles of librarians, highlighting the potential for a new and valuable profession in the form of prompt engineering librarians. This innovative concept could bridge the gap in AI literacy and facilitate more effective interactions with AI systems, contributing to the broader goal of AI accessibility and understanding.

show abstract

“…GPT-3). Aside from methods that make reasoning explicit in a linear chain manner, another set of work produce recursive structures of reasoning, through either backward chaining Jung et al, 2022) or forward chaining (Bostrom et al, 2022). Our work contributes to this line of research, yet we depart from prior work by presenting the first approach that learns to generate relevant knowledge without requiring human-labeled gold knowledge.…”

Section: Related Workmentioning

confidence: 99%

Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering

Liu¹,

Skyler²,

Lu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Knowledge underpins reasoning. Recent research demonstrates that when relevant knowledge is provided as additional context to commonsense question answering (QA), it can substantially enhance the performance even on top of state-of-the-art. The fundamental challenge is where and how to find such knowledge that is high quality and on point with respect to the question; knowledge retrieved from knowledge bases are incomplete and knowledge generated from language models are inconsistent.We present RAINIER 1 , or Reinforced Knowledge Introspector, that learns to generate contextually relevant knowledge in response to given questions. Our approach starts by imitating knowledge generated by GPT-3, then learns to generate its own knowledge via reinforcement learning where rewards are shaped based on the increased performance on the resulting question answering. RAINIER demonstrates substantial and consistent performance gains when tested over 9 different commonsense benchmarks: including 5 datasets that are seen during model training, as well as 4 datasets that are kept unseen. Our work is the first to report that knowledge generated by models that are orders of magnitude smaller than GPT-3, even without direct supervision on the knowledge itself, can exceed the quality of commonsense knowledge elicited from GPT-3.

show abstract

Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations

Cited by 12 publications

References 0 publications

Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes

Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes

The prompt engineering librarian

Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering

Contact Info

Product

Resources

About