Multisensory Integration: The Case of a Time Window of Gesture–Speech Integration

Abstract

This experiment investigates the integration of gesture and speech from a multisensory perspective. In a disambiguation paradigm, participants were presented with short videos of an actress uttering sentences like "She was impressed by the BALL, because the GAME/DANCE...." The ambiguous noun (BALL) was accompanied by an iconic gesture fragment containing information to disambiguate the noun toward its dominant or subordinate meaning. We used four different temporal alignments between noun and gesture fragment: the identification point (IP) of the noun was either prior to (+120 msec), synchronous with (0 msec), or lagging behind the end of the gesture fragment (-200 and -600 msec). ERPs triggered to the IP of the noun showed significant differences for the integration of dominant and subordinate gesture fragments in the -200, 0, and +120 msec conditions. The outcome of this integration was revealed at the target words. These data suggest a time window for direct semantic gesture-speech integration ranging from at least -200 up to +120 msec. Although the -600 msec condition did not show any signs of direct integration at the homonym, significant disambiguation was found at the target word. An explorative analysis suggested that gesture information was directly integrated at the verb, indicating that there are multiple positions in a sentence where direct gesture-speech integration takes place. Ultimately, this would implicate that in natural communication, where a gesture lasts for some time, several aspects of that gesture will have their specific and possibly distinct impact on different positions in an utterance.

Speaker's hand gestures modulate speech perception through phase resetting of ongoing neural oscillations

“…Note for instance, that the PLV time profiles peak at word onset, showing a precise timing of the gesture effect on auditory speech. This temporal pattern coincides well with studies that investigated the time window of gesture-speech integration from an ERP perspective (Habets et al, 2011;Obermeier et al, 2011;Obermeier & Gunter, 2014). Moreover, please note that the two relevant conditions in our study (words with gesture and words without gesture, in the AV modality) contained body motion visual information (the upper part of the speaker's body was visible both conditions), with the only difference being the presence / absence of a hand beat gesture.…”

“…Previous ERP studies have investigated this temporal window of integration between concurrent gestures describing an object (i.e. iconic gestures) and speech (Habets et al, 2011;Obermeier, Holle & Gunter, 2011;Obermeier & Gunter, 2014). For instance, Obermeier & Gunter (2014) showed that the ERP signature of semantic integration between gesture and speech was affected by incongruence when an approximate temporal overlap between the gesture fragment and its affiliate word was maintained (between -200ms to +120ms around identification point of word).…”

“…iconic gestures) and speech (Habets et al, 2011;Obermeier, Holle & Gunter, 2011;Obermeier & Gunter, 2014). For instance, Obermeier & Gunter (2014) showed that the ERP signature of semantic integration between gesture and speech was affected by incongruence when an approximate temporal overlap between the gesture fragment and its affiliate word was maintained (between -200ms to +120ms around identification point of word). However, one must note that contrary to the gestures used by Obermeier & Gunter (2014), beat gestures do not convey semantic content, and therefore the temporal relationship between the apex and the target word might be more important.…”

Motion capture-based animated characters for the study of speech–gesture integration

“…Studies using electrophysiological measures of brain activity and event-related potentials (ERPs), that is more implicit measures of processing, have shown that semantic integration is affected by advancing gestures by more than 200 ms (Habets et al, 2011;Obermeier & Gunter, 2014).…”

“…Some studies record actors performing scripted gestures (e.g., Cassell, McNeill, & McCullough, 1999;Woodall & Burgoon, 1981). Others use video editing, combining different image sequences with the same audio track (e.g., Habets et al, 2011;Leonard & Cummins, 2011;Obermeier & Gunter, 2014), typically examining one gesture in isolation. This approach often requires the speaker's face to be masked to avoid distraction from asynchronous speech and lip movement.…”