Using cognitive psychology to understand GPT-3

Binz, Marcel; Schulz, Eric

doi:10.31234/osf.io/6dfgk

Cited by 28 publications

(25 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…One can view this as a form of weak supervision, where we do not force the top-level GWT representations to be human-specified labels, only that there is a simple relationship between these representations and utterances which humans would often associate with the corresponding meaning. Our discussion about causality should also suggest that passive observation may be insufficient: in order to capture the causal structure understood by humans, it may be necessary for learning agents to be embedded in an environment in which they can act and thus discover its causal structure [188,189]. Studying this kind of set-up was the motivation for our work on the Baby AI environment [130].…”

Section: High-level Representations Describe Verbalizable Conceptsmentioning

confidence: 99%

Inductive biases for deep learning of higher-level cognition

2022

View full text Add to dashboard Cite

A fascinating hypothesis is that human and animal intelligence could be explained by a few principles (rather than an encyclopaedic list of heuristics). If that hypothesis was correct, we could more easily both understand our own intelligence and build intelligent machines. Just like in physics, the principles themselves would not be sufficient to predict the behaviour of complex systems like brains, and substantial computation might be needed to simulate human-like intelligence. This hypothesis would suggest that studying the kind of inductive biases that humans and animals exploit could help both clarify these principles and provide inspiration for AI research and neuroscience theories. Deep learning already exploits several key inductive biases, and this work considers a larger list, focusing on those which concern mostly higher-level and sequential conscious processing. The objective of clarifying these particular principles is that they could potentially help us build AI systems benefiting from humans’ abilities in terms of flexible out-of-distribution and systematic generalization, which is currently an area where a large gap exists between state-of-the-art machine learning and human intelligence.

show abstract

Section: High-level Representations Describe Verbalizable Conceptsmentioning

confidence: 99%

Inductive biases for deep learning of higher-level cognition

2022

View full text Add to dashboard Cite

show abstract

“…Our work stands in contrast to the prevailing research trend of studying humanlike aspects of LLMs, which views each LLM as a single entity with its own knowledge base (Petroni et al 2019), its own ability to pass as human (e.g., Uchendu et al 2021), and its own personality (Karra, Nguyen, and Tulabandhula 2022). Binz and Schulz (2022) use cognitive psychology to address the question of whether LLMs "learn and think like people." Jones and Steinhardt (2022) use human cognitive biases, such as anchoring and framing effects, to evaluate an LLM's "errors" where it deviates from rational behavior.…”

Section: Related Workmentioning

confidence: 99%

Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies

Aher¹,

Arriaga²,

Kalai³

2022

Preprint

View full text Add to dashboard Cite

We propose a method for using a large language model, such as GPT-3, to simulate responses of different humans in a given context. We test our method by attempting to reproduce well-established economic, psycholinguistic, and social experiments. The method requires prompt templates for each experiment. Simulations are run by varying the (hypothetical) subject details, such as name, and analyzing the text generated by the language model. To validate our methodology, we use GPT-3 to simulate the Ultimatum Game, garden path sentences, risk aversion, and the Milgram Shock experiments. In order to address concerns of exposure to these studies in training data, we also evaluate simulations on novel variants of these studies. We show that it is possible to simulate responses of different people and that their responses are largely consistent with prior human studies from the literature. Using large language models as simulators offers advantages but also poses risks. Our use of a language model for simulation is contrasted with anthropomorphic views of a language model as having its own behavior.

show abstract

“…Currently, researchers are using numerous theories and tools to investigate these language systems. For example, Binz and Schulz (2022) use cognitive psychology to investigate GPT‐3. This paper used the well‐known psychology text analysis software LIWC – 2015 (Language Inquiry and Word Count) to review LaMDA’s responses provided as part of the Lemoine (2022) interview.…”

Section: Liwc Text Analysis Of Lamda’s Dialoguementioning

confidence: 99%

Massive data language models and conversational artificial intelligence: Emerging issues

O’Leary

2022

Intell Sys Acc Fin Mgmt

View full text Add to dashboard Cite

Google's LaMDA, Open AI's GPT-3, and Meta's BlenderBot are artificial intelligence (AI)-based chatbots, that have been trained on billions of documents creating the notion of "massive data." These systems use human-generated documents to capture words and relationships between words that people use when they communicate. This paper examines some of the similarities of these systems and the emerging issues regarding these massive data language models, including whether they are sentient, the use and impact of scale, information use and ownership, and explanations of discussions and answers. This paper also directly investigates some artifacts of Google's LaMDA and compares them with Meta's BlenderBot. Finally, this paper examines emerging issues and questions deriving from our analysis.

show abstract

Using cognitive psychology to understand GPT-3

Cited by 28 publications

References 40 publications

Inductive biases for deep learning of higher-level cognition

Inductive biases for deep learning of higher-level cognition

Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies

Massive data language models and conversational artificial intelligence: Emerging issues

Contact Info

Product

Resources

About