Decoding and encoding models reveal the role of mental simulation in the brain representation of meaning

Abstract

How the brain representation of conceptual knowledge vary as a function of processing goals remains unclear. We hypothesized that the brain representation of semantic categories is shaped by the depth of processing. Participants were presented with visual words during functional MRI. During shallow processing, participants had to read the items. During deep processing, they had to mentally simulate the features associated with the words. Multivariate classification, informational connectivity and encoding models were used to reveal how the depth of processing determines the brain representation of word meaning. Decoding accuracy in putative substrates of the semantic network was enhanced when the depth processing was high, and the brain representations were more generalizable in semantic space relative to shallow processing contexts. This pattern was observed even in association areas in inferior frontal and parietal cortex. Deep information processing during mental simulation also increased the informational connectivity within key substrates of the semantic network. To further examine the properties of the words encoded in brain activity, we compared computer vision models -associated with the image referents of the words -and word embeddings. Computer vision models explained more variance of the brain responses across multiple areas of the semantic network. These results indicate that the brain representation of word meaning is highly malleable by the depth of processing imposed by the task, relies on access to visual representations and is highly distributed, including prefrontal areas previously implicated in semantic control.

Neurocognitive mechanisms supporting the generalization of concepts across languages

“…However, the decoding performance was significantly better in deep compared to shallow processing in IPL, VTL, and PCG for Spanish and IPL, VTL, and IFG for Basque. The superior decoding performance in the deep relative to shallow processing condition aligns with other recent observations in our laboratory (Soto, Sheikh, Mei, & Santana, 2019) and indicates that the task requirement had an impact on the brain representation of meaning.…”

“…In our study, word meaning could be decoded from patterns of activity in inferior frontal gyrus, namely, pars opercularis and pars triangularis, both within-, and also cross-languages. These results implicate this region in semantic representation (see also (Buchweitz et al, 2012;Shinkareva et al, 2011;Soto et al, 2019)). It is typically assumed that bilinguals are constantly switching between the two languages, selecting one and inhibiting the other based on task goals.…”

“…This pattern of results indicates that cross-language generalization in the deep processing case is not merely due to the increased signal to noise ratio of the multivoxel patterns corresponding to living and non-living items or merely based on modality-specific representations triggered by mental (e.g. visual) imagery processes occurring more strongly during deep relative to shallow processing (Soto et al, 2019). Our results also indicate that language-independent neural representations of semantic knowledge may not be easily generated during bottom-up information processing (i.e.…”