In-Context Analogical Reasoning with Pre-Trained Language Models

Hu, Xuesong; Storks, Shane; Lewis, Richard L.; Chai, Joyce

doi:10.18653/v1/2023.acl-long.109

Cited by 4 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, they mostly evaluate word analogies or simple analogies between two sentences, leaving complex structures in analogical reasoning unstudied. Webb et al (2022) and Hu et al (2023) examine the abstract language-based analogy task and evaluate the analogical reasoning ability of LLMs on this task. Compared to their task, our task requires the intensive involvement of commonsense, encyclopedic, and cultural (e.g., idiom and historical) knowledge.…”

Section: Related Workmentioning

confidence: 99%

Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction

Yuan,

Chen,

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

The vital role of analogical reasoning in human cognition allows us to grasp novel concepts by linking them with familiar ones through shared relational structures. Despite the attention previous research has given to word analogies, this work suggests that Large Language Models (LLMs) often overlook the structures that underpin these analogies, raising questions about the efficacy of word analogies as a measure of analogical reasoning skills akin to human cognition. In response to this, our paper introduces a task of analogical structure abduction, grounded in cognitive psychology, designed to abduce structures that form an analogy between two systems. In support of this task, we establish a benchmark called SCAR, containing 400 scientific analogies from 13 distinct fields, tailored for evaluating analogical reasoning with structure abduction. The empirical evidence underlines the continued challenges faced by LLMs, including ChatGPT and GPT-4, in mastering this task, signifying the need for future exploration to enhance their abilities. 1

show abstract

Section: Related Workmentioning

confidence: 99%

Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction

Yuan,

Chen,

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

show abstract

“…Remarkable for its ability to generate human-like text, ChatGPT has found extensive use in a diverse range of applications, spanning from marketing and reasoning to content generation [4][5][6][7][8][9][10][11] . Furthermore, ChatGPT offers a unique platform to evaluate the existence and manifestation of cognitive biases traditionally associated with human subjects.…”

mentioning

confidence: 99%

Is ChatGPT Humanly Irrational?

Ma,

Zhang,

Saunders

2023

Preprint

View full text Add to dashboard Cite

We delve into the fascinating crossroads of artificial intelligence (AI) and cognitive science, spotlighting the OpenAI advanced language model, ChatGPT. Renowned for generating human-like text, ChatGPT has been widely used in various applications. However, its ability to replicate human cognitive processes, particularly decision-making behavior, remains largely unexplored and untapped. We evaluate ChatGPT's decision-making patterns and show that they strikingly mirror those of human subjects, even patterns traditionally termed ''irrational'' under standard economic theory. This finding challenges the prevailing assumption that AI systems operate solely on rational computations. It suggests that, despite its algorithmic nature, ChatGPT can reflect human cognitive biases when simulating human roles, thus adding a new dimension to our understanding of AI behaviour. Our result places AI models like ChatGPT in a broader context of cognitive science, indicating their potential to mimic not just human language but also human cognitive processes. From a broader perspective, our findings underscore the capacity of AI in behavioral research and stimulate a necessary dialogue on AI design, transparency, and ethical implications. Our study bridges human and machine intelligence, highlighting the potential of AI to enhance our understanding of decision-making processes in artificial agents.

show abstract