Decoding and encoding models reveal the role of mental simulation in the brain representation of meaning

Abstract

How the brain representation of conceptual knowledge varies as a function of processing goals, strategies and task-factors remains a key unresolved question in cognitive neuroscience. In the present functional magnetic resonance imaging study, participants were presented with visual words during functional magnetic resonance imaging (fMRI). During shallow processing, participants had to read the items. During deep processing, they had to mentally simulate the features associated with the words. Multivariate classification, informational connectivity and encoding models were used to reveal how the depth of processing determines the brain representation of word meaning. Decoding accuracy in putative substrates of the semantic network was enhanced when the depth processing was high, and the brain representations were more generalizable in semantic space relative to shallow processing contexts. This pattern was observed even in association areas in inferior frontal and parietal cortex. Deep information processing during mental simulation also increased the informational connectivity within key substrates of the semantic network. To further examine the properties of the words encoded in brain activity, we compared computer vision models—associated with the image referents of the words—and word embedding. Computer vision models explained more variance of the brain responses across multiple areas of the semantic network. These results indicate that the brain representation of word meaning is highly malleable by the depth of processing imposed by the task, relies on access to visual representations and is highly distributed, including prefrontal areas previously implicated in semantic control.

Distinct roles for the anterior temporal lobe and angular gyrus in the spatiotemporal cortical semantic network

“…Second, from a cognitive perspective, we must ask why the hub region of effective connectivity changes over time from ATL to AG even though the modulation of neural activity remains localized to the ATL? The inferior parietal lobe (IPL) in general and AG in particular are well known to be involved in higher cognitive function, not only for semantics ( Binder et al 2009 ; Bonner et al 2013 ; Hartwigsen et al 2016 ; Kuhnke et al 2020 ; Soto et al 2020 ), but also for broader cognition such as episodic memory and executive functions for complex task demands ( Humphreys et al 2015 ; Humphreys and Lambon Ralph 2015 ; Bonnici et al 2016 ; Tibon et al 2019 ). Additionally, as a key node of the brain’s default mode network, IPL has been suggested as a multi-purpose hub in the brain ( Buckner et al 2009 ; Seghier 2012 ; Raichle 2015 ), and has been shown to be affected by brain diseases such as Alzheimer’s and dementia ( Buckner et al 2008 ; Spreng et al 2010 ).…”

Multimodal neurocognitive markers of frontal lobe epilepsy: Insights from ecological text processing

“…Following machine-learning analysis guidelines ( Dobbin and Simon, 2011 ), we split the datasets in a ratio of 80% for training, and 20% for testing, using random division, to test for generalizability without employing the testing dataset during the validation phase for out-of-folds predictions (for details, see Supplementary data 5 ). The 80/20 split is the gold-standard for obtaining robust cross-validation results across fields ( Poldrack et al, 2017 ), including neuroimaging research (e.g., ( Lanka et al, 2020 )), in general, and neurolinguistic studies (e.g., ( Soto et al, 2020 )), in particular. We trained the model with all the set of normalized features (i.e., verb-related and circumstantial information outcomes in each text type, FA results for the 10 JHU atlas tracts, and results for each of the seeds in each of the rsFC networks).…”

Decoding social knowledge in the human brain

“…Our results suggest that both the ACC and Ins, as part of the salience network (Uddin 2015), may be involved in representing these distinct aspects of social knowledge related to the likableness and affect dimensions. Notably, this was observed in a context that did not require participants to perform overt responses to items in a social experimental task setting, rather we only required mental simulation of social situations based on personal, idiosyncratic experiences (Soto et al 2020). These findings need further exploration in future studies aimed specifically to better understand the role of the ACC and Ins in the simulation of social contexts.…”

“…Originally, theoretical models suggested that the brain represents concepts as amodal symbols (Fodor 1975(Fodor , 1998. A more recent approach argues that conceptual representations are grounded in the sensorimotor experiences associated with them (Barsalou 1999;Prinz 2004), which can be re-enacted via mental simulation processes (Soto et al 2020). This grounded cognition framework was initially conceived the study of concrete concepts.…”