The role of mental simulation in primate physical inference abilities

Abstract

Primates can richly parse sensory inputs to infer latent information, and adjust their behavior accordingly. It has been hypothesized that such flexible inferences are aided by simulations of internal models of the external world. However, evidence supporting this hypothesis has been based on behavioral models that do not emulate neural computations. Here, we test this hypothesis by directly comparing the behavior of humans and monkeys in a ball interception task to that of recurrent neural network (RNN) models with or without the capacity to “simulate” the underlying latent variables. Humans and monkeys had strikingly similar behavioral patterns suggesting common underlying neural computations. Comparison between primates and a large class of RNNs revealed that only RNNs that were optimized to simulate the position of the ball were able to accurately capture key features of the behavior such as systematic biases in the inference process. These results are consistent with the hypothesis that primates use mental simulation to make flexible inferences. Moreover, our work highlights a general strategy for using model neural systems to test computational hypotheses of higher brain function.

Parametric control of flexible timing through low-dimensional neural manifolds

“…Our analyses suggest that low-dimensional networks learn simple parametric models of the dynamics, where the relevant parameter is controlled by a contextual cue. We expect these results to extend to other parametric forms of generalization (Rajalingham et al, 2021), and provide a fundamental building block for neural networks that implement more complex internal models of the external world.…”

Invariant representation of physical stability in the human brain

“…Instead, it has been proposed that a general ability to predict what will happen next in physical scenarios will require a more structured representation of the physical world that will support forward simulation ( Battaglia et al, 2013 ; Ullman et al, 2017 ). A parallel debate is raging in cognitive science ( Lerer et al, 2016 ; Conwell et al, 2019 ; Battaglia et al, 2013 ; Firestone and Scholl, 2017 ; Firestone and Scholl, 2016 ; Ludwin-Peery et al, 2019 ; Davis and Marcus, 2015 ; Chater and Oaksford, 2017 ; Davis et al, 2017 ; Ludwin-Peery et al, 2020 ), between those who argue that because human physical inferences occur rapidly ( Firestone and Scholl, 2017 ) and preattentively ( Firestone and Scholl, 2017 ) they are computed by something like a pattern recognition process, versus those who argue that human and primate physical inference behavior is best accounted for by mental simulation ( Battaglia et al, 2013 ; Ullman et al, 2017 ; Zhang et al, 2016 ; Gerstenberg et al, 2017 ; Rajalingham et al, 2021 ). Three lines of evidence from the present study indicate that pattern recognition alone – as instantiated in feedforward CNNs and the ventral visual pathway – is unlikely to explain physical inference in humans, at least for the case of physical stability.…”

Invariant representation of physical stability in the human brain

“…Instead, it has been proposed that a general ability to predict what will happen next in physical scenarios will require a more structured representation of the physical world that will support forward simulation. 8,9 A parallel debate is raging in cognitive science 5,6,8,10,17,[31][32][33][34]35 , between those who argue that because human physical inferences occur rapidly 10 and pre-attentively 10 they are computed by something like a pattern recognition process, versus those who argue that human and primate physical inference behavior is best accounted for by mental simulation 8,9,11,12,36 . Three lines of evidence from the present study support the simulation view.…”