Does audiovisual speech perception use information about facial configuration?

Abstract

The McGurk illusion eVectively demonstrates the audiovisual nature of speech perception. When an auditory syllable is dubbed onto an incongruous visual syllable the resulting percept is usually not either of the components, but their combination or fusion. The present experiment investigated the persistence of the McGurk eVect when the facial con guration context of the audiovisual stimuli was manipulated. Two congruent and two incongruent audiovisual syllables were created from spoken /ipi/ and /iki/. These audiovisual tokens were uttered by seven facial con gurations of ve talkers. All facial con gurations produced a clear McGurk eVect (reported /iti/ for heard /ipi/ + seen /iki/), but the eVect was signi cantly less when uttered by an asymmetrically scrambled face con guration. The results also showed signi cant diVerences in the persistence of the McGurk eVect between diVerent talkers. In sum, facial con gural information can be used in the audiovisual integration of certain speech segments. This information is not necessary for the integration to occur, but the integration process can be disrupted by face stimuli violating the normal con gural information.

Audiovisual speech from emotionally expressive and lateralized faces

“…However, research into face and speech perception has challenged this notion, demonstrating that configural distortions to the face can degrade its efficacy for supporting visual speech detection (Hietanen et al, 2001;Rosenblum et al, 2000). The results of the current research seem to be more consistent with these latter studies, suggesting that affect may also interact with speech.…”

“…While the affective expressions clearly influenced performance, the large configural distortions created by visual manipulations had little influence on audiovisual speech detection, only doing so inasmuch as they interacted with the affective displays. Hence, while previous research demonstrated that some methods of scrambling faces can influence audiovisual syllable detection (Hietanen et al, 2001;Rosenblum et al, 2000), in the current experiments, with full-length sentences, perceivers were found to suffer little audiovisual speech decrement, if any, from our more subtle configural distortions. The facial chimeras are unusual in configuration in that they are perfectly symmetrical, and despite efforts to maintain a centralized head position for the actor, minor head shifts were normal (thus producing small changes in the head size of her image).…”

“…In experiments that distort the configuration of a talking face (by inversion of facial features and repositioning features within a face), the decrements were found to disrupt the McGurk effect, a powerful speech illusion (Hietanen et al, 2001;Rosenblum et al, 2000). Briefly, the McGurk effect entails pairing a spoken auditory syllable with a discrepant visual syllable in an audiovisual speech token.…”

Top-down and bottom-up modulation of audiovisual integration in speech

“…It was, however, not face inversion in itself that was responsible for diminishing the McGurk effect but rather the inversion of the lips, which would result in a loss of configural coherence between the articulatory movements and the auditory signal (Rosenblum, Yakel & Green, 2000). Hietanen, Manninen, Sams and Surakka (2001) confirmed that, provided lips are presented upright, a normal face configuration was not necessary for a McGurk illusion to occur. They found that only an asymmetrically scrambled face stimulus (both eyes, mouth, and noise being horizontally and vertically displaced) produced strong deterioration.…”

Effects of sentence context and expectation on the McGurk illusion

“…Crucially, these procedures manipulated only subjectspecific cognitive variables (Engel et al, 2001), without ever changing the bottom-up input signal (i.e., the auditory or the visual component of the McGurk stimuli, as, e.g., in MacDonald et al, 2000), the immediate sensory context (as in Green & Gerdeman, 1995;Hietanen, Manninen, Sams, & Surakka, 2001;Walker et al, 1995), or the attentional weighting of the auditory vs. visual input signals (as in Massaro & Warner, 1977;Summerfield & McGrath, 1984).…”