Invariant representation of physical stability in the human brain

Abstract

Successful engagement with the world requires the ability to predict what will happen next. Here, we investigate how the brain makes a fundamental prediction about the physical world: whether the situation in front of us is stable, and hence likely to stay the same, or unstable, and hence likely to change in the immediate future. Specifically, we ask if judgments of stability can be supported by the kinds of representations that have proven to be highly effective at visual object recognition in both machines and brains, or instead if the ability to determine the physical stability of natural scenes may require generative algorithms that simulate the physics of the world. To find out, we measured responses in both convolutional neural networks (CNNs) and the brain (using fMRI) to natural images of physically stable versus unstable scenarios. We find no evidence for generalizable representations of physical stability in either standard CNNs trained on visual object and scene classification (ImageNet), or in the human ventral visual pathway, which has long been implicated in the same process. However, in frontoparietal regions previously implicated in intuitive physical reasoning we find both scenario-invariant representations of physical stability, and higher univariate responses to unstable than stable scenes. These results demonstrate abstract representations of physical stability in the dorsal but not ventral pathway, consistent with the hypothesis that the computations underlying stability entail not just pattern classification but forward physical simulation.

Cognition through internal models: Mirror neurons as one manifestation of a broader mechanism

“…( 3, 10 ) These models play a ubiquitous role, providing a common substrate to inform our perception, cognition, imagination, and planning. The neural basis for this cognition through internal models framework remains largely unexplored, though preliminary neuroimaging evidence points to a role for PPC ( 10, 25 ). Our results, demonstrating shared representations that can be decomposed into basic building blocks, support this computational architecture, and provide preliminary insight into its neural implementation within human PPC.…”

Object representation in a gravitational reference frame

“…Thus, while orientation tuning of ∼35% of CIP neurons was sensitive to monkey tilt and gravity-aligned information could be extracted with a neural network 41 , there was no explicit tuning in a gravitational reference frame or dominance of gravitational information as found here. There is however compelling human fMRI evidence that parietal and frontal cortex are deeply involved in perceiving and predicting physical events 42 , and have unique abstract signals for stability not detected in ventral pathway 43 (though these could reflect decision-making processes 44, 45 ). Our results and others 46–48 suggest nonetheless that ventral pathway object and scene processing may be a critical source of information about gravity and its effects on objects, especially when detailed object representations are needed to assess precise shape, structure, support, strength, flexibility, compressibility, brittleness, specific gravity, mass distribution, and mechanical features to understand real world physical situations.…”

From infancy, we model people as minds, acting in a physical world

“…The proposal that infants hold an intuitive theory of psychology that operates over a world model has implications for brain function and development. This joint reasoning over psychology and physics is plausibly supported, in adult brains, by regions engaged in social and physical reasoning (Amodio & Frith, 2006;Deen et al, 2020;DiNicola et al, 2020;Gao et al, 2012;Isik et al, 2017;Pramod et al, 2021;Saxe & Wexler, 2005;Schwettmann et al, 2019;Walbrin et al, 2018), as well as regions for recognizing and perceiving instrumental action (Caramazza et al, 2014;de Lange et al, 2008;Iacoboni et al, 2005;Pobric & Hamilton, 2006). In infants, similar regions respond preferentially to social and physical stimuli (Deen et al 2017;Grossmann 2015;Hyde et al 2018;Kosakowski et al 2021;Lloyd-Fox et al 2009;Powell et al 2017;Wilcox and Biondi 2015).…”

“…Do regions like the superior temporal sulcus (STS) and medial prefrontal cortex (MPFC), that preferentially respond to social stimuli regardless of modality (stories read aloud or signed; videos and images of people; sounds like laughter and speech) (Deen et al, 2015(Deen et al, , 2020DiNicola et al, 2020;Dodell-Feder et al, 2011;Gao et al, 2012;Isik et al, 2017;Koster-Hale et al, 2017;Richardson et al, 2020;Walbrin et al, 2018) contain information not only about agents and their mental states, but also about their physical constraints? Do regions like supramarginal gyri and superior parietal cortices, that are preferentially engaged when people make physical predictions and judgments (Fischer et al, 2016;Mason & Just, 2016;Newman et al, 2005;Pramod et al, 2022;Schwettmann et al, 2019;van Nuenen et al, 2012), contain information used for computing what agents want and know? The tools of cognitive neuroscience also allow us to gain conceptual clarity on the behaviors we measure in human infants.…”