The Predictive Power of Spatial Relational Reasoning Models: A New Evaluation Approach

Abstract

In the last few decades, cognitive theories for explaining human spatial relational reasoning have increased. Few of these theories have been implemented as computational models, however, even fewer have been compared computationally to each other. A computational model comparison requires, among other things, a still missing quantitative benchmark of core spatial relational reasoning problems. By presenting a new evaluation approach, this paper addresses: (1) developing a benchmark including raw data of participants, (2) reimplementation, adaptation, and extension of existing cognitive models to predict individual responses, and (3) a thorough evaluation of the cognitive models on the benchmark data. The paper shifts the research focus of cognitive modeling from reproducing aggregated response patterns toward assessing the predictive power of models for the individual reasoner. It demonstrate that not all psychological effects can discern theories. We discuss implications for modeling spatial relational reasoning.

Modelling the Accuracy Rates of Spatial Relational Reasoning Problems: An Analysis Facilitating ACT-R and PRISM Theory

“…Typically, reasons for the difficulty of a problem are explained by the number of viable alternative mental models and demands placed on working memory required to solve the problem. Related theories such as the preferred model theory (Knauff, Rauh, & Scheller, 1995;Jahn et al, 2008;Ragni, & Knauff, 2013;Ragni, Brand & Riesterer, 2021) attempt to explain how human cognition is biased toward constructing specific mental models over others and the implications that come from these preferences. In preferred inferences in reasoning with spatial mental models (PRISM), Ragni and Knauff (2013) present an implementation of preferred model theory through a computational model simulating how such mental models are cognitively constructed, inspected, and varied to make inferences for relational reasoning problems.…”

“…In contrast to the mental model theory, related theories such as the preferred model theory (Knauff et al, 1995;Jahn et al, 2008;Ragni & Knauff, 2013;Ragni, Brand & Riesterer, 2021) argue that people only construct a single mental model in most situations and remain almost blind to other interpretations unless explicitly told to acknowledge alternatives. In preferred model theory, the three stages of comprehension, description, and validation are renamed to better characterise what occurs in each respective stage as model generation, inspection, and variation phases (Ragni & Knauff, 2013).…”

“…The previously mentioned PRISM is one example of an attempt to model relational reasoning problems as more computationally explicit. PRISM is a computational description (Ragni & Knauff, 2013) of the preferred mental model theory which can be implemented (Ragni et al, 2021) to simulate and explain how preferences are found in the construction, inspection, and variation phases of human reasoning with mental models. Two descriptions can broadly characterise the construction and inspection phases of PRISM models.…”

“…Four Python ACT-R models of spatial relational reasoning were created based on preferred inferences in reasoning with spatial mental models (PRISM) (Boeddinghaus et al, 2006;Ragni & Knauff, 2013;Ragni et al, 2021). Each model is based on a single question taken from the Multidimensional Relational Reasoning Task (MRRT) Cortes et al, (2021).…”

“…PRISM is an implementation of the preferred mental model theory and is intended to simulate human performance on spatial reasoning tasks (Ragni, & Knauff, 2013;Ragni et al, 2021). The constructed ACT-R models are based on PRISM's construction phase and inspection phase, which conceptualizes spatial memory representations as two-dimensional arrays.…”