A Data Source for Reasoning Embodied Agents

Lanchantin, Jack; Sukhbaatar, Sainbayar; Synnaeve, Gabriel; Sun, Yueqiang; Srinet, Kavya; Szlam, Arthur

doi:10.1609/aaai.v37i7.26017

Cited by 4 publications

(3 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…CoT has been shown to significantly enhance performance on complex reasoning tasks such as arithmetic and commonsense reasoning. This success has spurred several subsequent works that adapt different strategies to improve CoT, including self-consistency , Least-to-Most prompting , self-notes (Lanchantin et al, 2023), Progressive-Hint Prompting , and selfpolish . Despite the remarkable success of CoT prompting, previous studies have primarily focused on how to use CoT to achieve the best results while ignoring how to construct prompting examples.…”

Section: Chain-of-thought Promptmentioning

confidence: 99%

See 1 more Smart Citation

INFORM : Information eNtropy based multi-step reasoning FOR large language Models

Zhou,

You,

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Large language models (LLMs) have demonstrated exceptional performance in reasoning tasks with dedicated Chain-of-Thought (CoT) prompts. Further enhancing CoT prompts with exquisite exemplars can significantly improve reasoning performance. However, the effectiveness of CoT prompts may fluctuate dramatically with different choices of in-context examples. Additionally, manual construction of rationale steps can be time-consuming, presenting challenges for the widespread adoption of CoT prompting. In this work, we propose a novel approach by introducing information entropy (IE) as a criteria on for CoT prompt selection. We extend this criterion to the CoT generation and inference stages, automatically generating CoT prompts with higher information entropy scores and adaptively determining the number of samples. These three stages together form our proposed information entropy based multi-step reasoning for large language models, named INFORM. Our experiments across seven reasoning benchmarks utilizing two language models(GPT-3.5-Turbo and text-davinci-003) demonstrate the superiority of INFORM both in performance and efficiency.

show abstract

Section: Chain-of-thought Promptmentioning

confidence: 99%

“…CoT has shown promising results in improving the reasoning abilities of LLMs. Various strategies like self-notes (Lanchantin et al, 2023), progressive-hint prompting and Least-to-Most prompting , have been proposed to enhance CoT further.…”

Section: Introductionmentioning

confidence: 99%

INFORM : Information eNtropy based multi-step reasoning FOR large language Models

Zhou,

You,

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…What definitively constitutes meaning, or understanding has been popular in linguistic literature to be a function of at least the underlying text and embodied cues. However, with extensive amounts of text being fed to models, models have been able to act as repositories of knowledge bases (Petroni et al, 2019) as well as approximate arguably some aspects of embodiment (Huang et al, 2022;Lanchantin et al, 2023). So, while one definitely can't discount Selbst et al ( 2019)'s recommendations that many of the contextual and politically contested topics should not be technology forced, LLMs do not seem completely handicapped for subjective tasks which require a high degree of uncertainty -For example, Thomas et al (2023) show how LLMs can be used to accurately model searcher preferences or when LLMs are used to replace human evaluations (Chiang and Lee, 2023) -tasks which generally require a lot of human annotation effort.…”

Section: The Solutionism Trapmentioning

confidence: 99%

Large Language Models as SocioTechnical Systems

Dhole

2023

Proceedings of the Big Picture Workshop

View full text Add to dashboard Cite

The expectation of Large Language Models (LLMs) to solve various societal problems has ignored the larger socio-technical frame of reference under which they operate. From a socio-technical perspective, LLMs are necessary to look at separately from other ML models as they have radically different implications in society never witnessed before. In this article, we ground Selbst et al. ( 2019)'s five abstraction traps -The Framing Trap, The Portability Trap, The Formalism Trap, The Ripple Effect Trap and the Solutionism Trap in the context of LLMs discussing the problems associated with the abstraction and fairness of LLMs. Through learnings from previous studies and examples, we discuss each trap that LLMs fall into, and propose ways to address the points of LLM failure by gauging them from a socio-technical lens. We believe the discussions would provide a broader perspective of looking at LLMs through a sociotechnical lens and our recommendations could serve as baselines to effectively demarcate responsibilities among the various technical and social stakeholders and inspire future LLM research.

show abstract