The Large-Scale Organization of Gestures and Words in the Middle Temporal Gyrus

Abstract

The middle temporal gyrus (MTG) has been shown to be recruited during the processing of words, but also during the observation of actions. Here we investigated how information related to words and gestures is organized along the MTG. To this aim, we measured the BOLD response in the MTG to video clips of gestures and spoken words in 17 healthy human adults (male and female). Gestures consisted of videos of an actress performing object-use pantomimes (iconic representations of object-directed actions; e.g., playing guitar), emblems (conventional gestures, e.g., thumb up), and meaningless gestures. Word stimuli (verbs, nouns) consisted of video clips of the same actress pronouncing words. We found a stronger response to meaningful compared with meaningless gestures along the whole left and large portions of the right MTG. Importantly, we observed a gradient, with posterior regions responding more strongly to gestures (pantomimes and emblems) than words and anterior regions showing a stronger response to words than gestures. In an intermediate region in the left hemisphere, the response was significantly higher to words and emblems (i.e., items with a greater arbitrariness of the sign-to-meaning mapping) than to pantomimes. These results show that the large-scale organization of information in the MTG is driven by the input modality and may also reflect the arbitrariness of the relationship between sign and meaning.Here we investigated the organizing principle of information in the middle temporal gyrus, taking into consideration the inputmodality and the arbitrariness of the relationship between a sign and its meaning. We compared the middle temporal gyrus response during the processing of pantomimes, emblems, and spoken words. We found that posterior regions responded more strongly to pantomimes and emblems than to words, whereas anterior regions responded more strongly to words than to pantomimes and emblems. In an intermediate region, only in the left hemisphere, words and emblems evoked a stronger response than pantomimes. Our results identify two organizing principles of neural representation: the modality of communication (gestural or verbal) and the (arbitrariness of the) relationship between sign and meanings.

The representational space of observed actions

“…At the same time, Hafri et al (2017) demonstrated that it is possible to decode actions across stimulus format in the posterior LOTC, in line with the view that action-related representations in this area does not necessarily require visual motion. Together, our findings suggest that this area captures semantic aspects of actions at higher-order visual levels, whereas anterior portions of the left LOTC might capture these aspects at stimulus-independent or verbal levels of representation (see also Lingnau and Downing, 2015 and Papeo et al, 2019).…”

“…Together, the results of the multiple regression RSA obtained for the different models are in line with the view that the LOTC hosts a variety of different, partially overlapping representations of action components that are likely to be integrated flexibly according to task demands (see also Lingnau and Downing, 2015). Multiple, possibly overlapping basic dimensions have been proposed to underlie the organization of these different action components within the LOTC, among them the input modality (visual versus non-visual; Lingnau and Downing, 2015; Papeo et al, 2019), the presence and orientation of a person, and the directedness of actions toward persons or other targets (Wurm et al, 2017; Wurm and Caramazza, 2019b).…”

Altered regional activity and connectivity of functional brain networks in congenital unilateral conductive hearing loss

“…These could be related to fewer resources of the audiovisual integration cortices allocated to visual cortices of the contralateral hemisphere in a resting state, with more resources allocated to enhance ipsilateral audiovisual processing. Interestingly, the intermediate region of the left middle temporal gyrus, which was found to be sensitive to words and emblems ( Papeo et al, 2019 ), demonstrated lower ALFF values in the UM group. This finding suggests that audiovisual plasticity may be region-specific, as different regions of visual processing (fusiform and middle temporal gyrus) differ in spontaneous regional activity in the resting state.…”

Intrinsic brain activity associated with eye gaze during mother–child interaction

“…The angular gyrus has a wide range of functional relationships with many brain regions and is called a connection hub because of its functionality 40 . Similarly, the precentral gyrus was suggested to be involved in executing voluntary motor movements 41 and the middle temporal gyrus to be involved in actions and word meaning processing 42 . As these areas are constituents of the CEN 41,42 , they may have induced over-focus on tasks and interference with eye contact.…”

“…Similarly, the precentral gyrus was suggested to be involved in executing voluntary motor movements 41 and the middle temporal gyrus to be involved in actions and word meaning processing 42 . As these areas are constituents of the CEN 41,42 , they may have induced over-focus on tasks and interference with eye contact. In relation to the quality of the mother-child interaction, the IRS score was positively associated with both eye gaze and eye contact in mothers, although it was only associated with eye contact in children.…”