Exploring the boundaries of gesture-speech integration during language comprehension

Abstract

The present review explores the integrated relationship between gesture and speech during language comprehension. Taking a broad view, it presents a conceptual framework that approaches the comprehension of gesture and speech along three different dimensions: (1) components of language (semantics, pragmatics, phonetics and syntax), (2) levels of analysis (social, cognitive, and biological) and (3) timeframes of integration (online, moment-to-moment, developmental). The evidence suggests that some linguistic components (e.g., concrete semantic and pragmatic) are deeply connected to gestures, but others (e.g., abstract semantic, syntactic and phonetic) are less so. In this way, the hands help to delineate what aspects of language function as part of the body and what aspects operate independently of it. 1. Of course, technically, all of language is tied to the body through its production and comprehension. The distinction here concerns the extent to which the physical constraints of the body-through the systems of perception and action-are a relatively small or large part of what fundamentally constitutes language (for more on this distinction in cognition generally, see Barsalou, 1999).

The Efficacy of Gesture on Second Language Pronunciation: An Exploratory Study of Handclapping as a Classroom Instructional Tool

“…Although gestures have been used in L2 classrooms for segmental instruction as well (e.g., Hudson, 2011; Roberge, Kimura, & Kawaguchi, 1996), their benefits do not seem to be as clear as those for suprasegmental instruction (see Kelly, 2017 for a review). One possibility for this dichotomy is that gesture and speech may be connected with higher level features such as suprasegmental information and semantic properties, but they may not be integrated at the lower levels of phoneme processing—making the positive effects of gestures less likely to manifest themselves in segmental instruction (Kelly, 2017).…”

“…Although gestures have been used in L2 classrooms for segmental instruction as well (e.g., Hudson, 2011; Roberge, Kimura, & Kawaguchi, 1996), their benefits do not seem to be as clear as those for suprasegmental instruction (see Kelly, 2017 for a review). One possibility for this dichotomy is that gesture and speech may be connected with higher level features such as suprasegmental information and semantic properties, but they may not be integrated at the lower levels of phoneme processing—making the positive effects of gestures less likely to manifest themselves in segmental instruction (Kelly, 2017). Evidence from experimental studies with Chinese phonemic tone contrasts have seemed to be in line with this suggestion: Virtually no benefit was found when L1 English participants repeated Chinese monosyllables after models with metaphoric gestures depicting the relative height and contour of tones (Zheng et al., 2018).…”

The Embodied Teaching of Spatial Terms: Gestures Mapped to Morphemes Improve Learning

“…Neuroscientific research shows that perceptual and lexical-semantic spatial information have a parallel organization in the brain (Göksun et al, 2013) and that simple gestures can make meaningful differences in how complex language is understood (Holle et al, 2012), however, the relationship between speech, gesture and language comprehension is complex. Some research suggests that under certain circumstances, for example when cognitive demands are high or skill level is low, gestures may disrupt comprehension (McNeil et al, 2000;Kelly, 2017). Gesture theory, as outlined in the GfCH, makes predictions about the supportive effects of gestures for learning, but how to best use gestures in L2 classrooms is under-researched, leaving many questions unanswered.…”

“…Iconic gestures, which have a "close formal relationship to the semantic content of speech" (McNeill, 1992: p. 12) have been shown to be beneficial, but there are different kinds of iconicity (Perniss and Vigliocco, 2014). How children mentally represent conceptual information changes over time (Kelly, 2017), suggesting that development might influence which gestures are most effective. Further, as the MRC concept suggests, a substantial amount of the information we use to determine meaning is not associated with a single lexical item (Foster, 2001;Knoeferle et al, 2010).…”

Emblem Gestures Improve Perception and Evaluation of Non-native Speech

“…The integrated relationship between speech and gesture in languageproduction has led many researchers to study how these two parts ofthe system work together during language comprehension (for reviews, see Hostetter, 2011 ; Kelly, 2017 ). Specifically testing McNeill’s theory, Kelly et al (2010) advanced the integrated systems hypothesis to show that that the semantic relationship between speech and gesture affect the accuracy and speed of language comprehension.…”