We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3’s decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3’s behavior is impressive: It solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multiarmed bandit task, and shows signatures of model-based reinforcement learning. Yet, we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. Taken together, these results enrich our understanding of current large language models and pave the way for future investigations using tools from cognitive psychology to study increasingly capable and opaque artificial agents.

Numerous researchers have put forward heuristics as models of human decision-making.However, where such heuristics come from is still a topic of ongoing debate. In this work, we propose a novel computational model that advances our understanding of heuristic decision-making by explaining how different heuristics are discovered and how they are selected. This model -called bounded meta-learned inference (BMI) -is based on the idea that people make environment-specific inferences about which strategies to use while being efficient in terms of how they use computational resources. We show that our approach discovers two previously suggested types of heuristics -one reason decision-making and equal weighting -in specific environments. Furthermore, the model provides clear and precise predictions about when each heuristic should be applied: knowing the correct ranking of attributes leads to one reason decision-making, knowing the directions of the attributes leads to equal weighting, and not knowing about either leads to strategies that use weighted combinations of multiple attributes. In three empirical paired comparison studies with continuous features, we verify predictions of our theory and show that it captures several characteristics of human decision-making not explained by alternative theories.

We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3's decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3's behavior is impressive: it solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multi-armed bandit task, and shows signatures of model-based reinforcement learning. Yet we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. These results enrich our understanding of current large language models and pave the way for future investigations using tools from cognitive psychology to study increasingly capable and opaque artificial agents.

Numerous researchers have put forward heuristics as models of human decision making. However, where such heuristics come from is still a topic of ongoing debates. In this work we propose a novel computational model that advances our understanding of heuristic decision making by explaining how different heuristics are discovered and how they are selected. This model, called bounded meta-learned inference, is based on the idea that people make environment-specific inferences about which strategies to use, while being efficient in terms of how they use computational resources. We show that our approach discovers two previously suggested types of heuristics -- one reason decision making and equal weighting -- in specific environments. Furthermore, the model provides clear and precise predictions about when each heuristic should be applied: knowing the correct ranking of attributes leads to one reason decision making, knowing the directions of the attributes leads to equal weighting, and not knowing about either leads to strategies that use weighted combinations of multiple attributes. This allows us to gain new insights on mixed results of prior empirical work on heuristic decision making. In three empirical paired comparison studies with continuous features, we verify predictions of our theory, and show that it captures several characteristics of human decision making not explained by alternative theories.

The Einstellung effect was first described by Abraham Luchins in his doctoral thesis published in 1942. The effect occurs when a repeated solution to old problems is applied to a new problem even though a more appropriate response is available. In Luchins' so-called water jar task, participants had to measure a specific amount of water using three jars of different capacities. Luchins found that subjects kept using methods they had applied in previous trials, even if a more efficient solution for the current trial was available: an Einstellung effect. Moreover, Luchins studied the different conditions that could possibly mediate this effect, including telling participants to pay more attention, changing the number of tasks, alternating between different types of tasks, as well as putting participants under time pressure. In the current work, we reconstruct and reanalyze the data of the various experimental conditions published in Luchins' thesis. We furthermore show that a model of resource-rational decision-making can explain all of the observed effects. This model assumes that people transform prior preferences into a posterior policy to maximize rewards under time constraints. Taken together, our reconstructive and modeling results put the Einstellung effect under the lens of modern-day psychology and show how resource-rational models can explain effects that have historically been seen as deficiencies of human problem-solving.