The role of semantic abstractness and perceptual category in processing speech accompanied by gestures

Abstract

Space and shape are distinct perceptual categories. In language, perceptual information can also be used to describe abstract semantic concepts like a “rising income” (space) or a “square personality” (shape). Despite being inherently concrete, co-speech gestures depicting space and shape can accompany concrete or abstract utterances. Here, we investigated the way that abstractness influences the neural processing of the perceptual categories of space and shape in gestures. Thus, we tested the hypothesis that the neural processing of perceptual categories is highly dependent on language context. In a two-factorial design, we investigated the neural basis for the processing of gestures containing shape (SH) and spatial information (SP) when accompanying concrete (c) or abstract (a) verbal utterances. During fMRI data acquisition participants were presented with short video clips of the four conditions (cSP, aSP, cSH, aSH) while performing an independent control task. Abstract (a) as opposed to concrete (c) utterances activated temporal lobes bilaterally and the left inferior frontal gyrus (IFG) for both shape-related (SH) and space-related (SP) utterances. An interaction of perceptual category and semantic abstractness in a more anterior part of the left IFG and inferior part of the posterior temporal lobe (pTL) indicates that abstractness strongly influenced the neural processing of space and shape information. Despite the concrete visual input of co-speech gestures in all conditions, space and shape information is processed differently depending on the semantic abstractness of its linguistic context.

Language and Cognition

“…Tilles and Fontanari ( 2013 ) examine reinforcement and inference in cross-situational word learning. Nagels et al ( 2013a ) indicate the role of semantic abstractness and perceptual category in processing speech accompanied by gestures. Zhong et al ( 2013 ) study a self-organizing pre-symbolic neural model representing sensorimotor information.…”

Linking language to the visual world: Neural correlates of comprehending verbal reference to objects through pointing and visual cues

“…iconic gestures, pantomimes, or metaphoric gestures) and concurrent speech is that an increase in semantic unification load leads to an increase in LIFG activation (cf. Andric and Small, 2012;Dick et al, 2014;Hagoort et al, 2009;Kircher et al, 2009;Nagels et al, 2013;Özyürek, 2014;Skipper et al, 2007;Willems et al, 2007Willems et al, , 2009). For instance, iconic gestures that are unrelated to concurrently perceived speech require additional processing compared to iconic gestures that relate to the concurrently presented speech because building a conceptual representation on the basis of the different streams of information is more effortful in the former compared to the latter case.…”

Dealing with Digits : Arithmetic, Memory and Phonology in Deaf Signers

“…Behavioural findings suggest that there is a closer relationship between semantic and phonological processing in signed compared to spoken languages (Marshall, Rowley, & Atkinsson, 2013). In hearing individuals, abstract semantic processing has been found in the anterior portions of Broca's areas and in BA46 (Nagels, Chatterjee, Kircher, & Straube, 2013;Poldrack et al, 1999). Thus, the phonologyrelated activation found anterior to Broca's area for deaf signers may reflect a shift in the relative balance of semantic and phonological processing in signed language in the left inferior frontal gyrus (IFG; Hagoort, 2005;Marshall et al, 2013;Rudner et al, 2013).…”