What Iconic Gesture Fragments Reveal about Gesture–Speech Integration: When Synchrony Is Lost, Memory Can Help

Abstract

The present series of experiments explores several issues related to gesture-speech integration and synchrony during sentence processing. To be able to more precisely manipulate gesture-speech synchrony, we used gesture fragments instead of complete gestures, thereby avoiding the usual long temporal overlap of gestures with their coexpressive speech. In a pretest, the minimal duration of an iconic gesture fragment needed to disambiguate a homonym (i.e., disambiguation point) was therefore identified. In three subsequent ERP experiments, we then investigated whether the gesture information available at the disambiguation point has immediate as well as delayed consequences on the processing of a temporarily ambiguous spoken sentence, and whether these gesture-speech integration processes are susceptible to temporal synchrony. Experiment 1, which used asynchronous stimuli as well as an explicit task, showed clear N400 effects at the homonym as well as at the target word presented further downstream, suggesting that asynchrony does not prevent integration under explicit task conditions. No such effects were found when asynchronous stimuli were presented using a more shallow task (Experiment 2). Finally, when gesture fragment and homonym were synchronous, similar results as in Experiment 1 were found, even under shallow task conditions (Experiment 3). We conclude that when iconic gesture fragments and speech are in synchrony, their interaction is more or less automatic. When they are not, more controlled, active memory processes are necessary to be able to combine the gesture fragment and speech context in such a way that the homonym is disambiguated correctly.

Visual Context Enhanced: The Joint Contribution of Iconic Gestures and Visible Speech to Degraded Speech Comprehension

“…This was also found by Obermeier et al (2011), who used a paradigm similar to that of Holle and Gunter (2007) to reveal that when there was no temporal overlap of a word and a gesture, and participants were not explicitly asked to attend to the gestures, speech-gesture integration did not occur. However, in a subsequent study, in which the same stimuli were presented in multitalker babble noise, listeners did incorporate the gestural information with the speech signal to disambiguate the meaning of the sentence.…”

“…Nevertheless, most studies on lip movements as a visual enhancement of speech have used stimuli that only showed the lips or lower half of the face (e.g., Callan et al, 2003;Ross et al, 2007;Schwartz et al, 2004) to eliminate influences from the rest of the face or body. This is similar to studies in the domain of gestural enhancement of speech in noise, where most studies block the face of the speaker, the mouth, or just show the torso of the speaker, to eliminate influences from visible speech (e.g., Holle et al, 2010;Obermeier et al, 2011Obermeier et al, , 2012.…”

“…As Holle et al (2010) noted, however, their study (and other studies, see e.g., Obermeier et al, 2011Obermeier et al, , 2012 only focused on one visual articulator in speech-related audiovisual integration, namely, iconic gestures. Other visual articulators, such as lip movements, were deliberately excluded from the stimuli by blocking the actor's face with a black mask.…”

“…Previous research has shown that semantic information from iconic gestures is indeed processed by listeners and that iconic gestures can affect language comprehension at behavioral and neural levels (e.g., Beattie & Shovelton, 1999a, 1999b, 2002Holle & Gunter, 2007;Holler, Kelly, Hagoort, & Özyürek, 2010;Holler, Shovelton, & Beattie, 2009;Holler et al, 2014;Kelly, Barr, Church, & Lynch, 1999;Kelly, Healey, Özyürek, & Holler, 2015;Obermeier, Holle, & Gunter, 2011; for a review, see Özyürek, 2014). For example, in an electroencephalography (EEG) study, Holle and Gunter (2007) showed participants videos of an actor who uttered a sentence while gesturing.…”

Multisensory Integration: The Case of a Time Window of Gesture–Speech Integration

“…This was recently demonstrated in two ERP experiments on the integration of iconic gestures and speech (gesture-word integration: Habets, Kita, Shao, Özyürek, & Hagoort, 2011; gesture-sentence integration : Obermeier, Holle, & Gunter, 2011). For instance, the study by Obermeier et al (2011) showed that the integration of gesture fragments and speech occurs in an automatic fashion when they were presented simultaneously. When this synchrony was lost because the gesture fragment preceded speech by 1000 msec (i.e., there was no temporal overlap between gesture fragment and lexical affiliate), effortful processing, induced via task instruction, was necessary for a successful integration.…”