Gaze behavior in audiovisual speech perception: The influence of ocular fixations on the McGurk effect

Paré, Martin; Richler, Rebecca C.; Hove, Martin ten; Munhall, Kevin G.

doi:10.3758/bf03194582

Cited by 70 publications

(90 citation statements)

References 51 publications

(65 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In a study using extended monologues, Vatikiotis-Bateson et al (1998) this is nonetheless consistent with the substantially increased median fixation durations on both the nose and mouth found in the current study. Buchan et al (2007) used emotionally expressive talking faces and also saw an increase in median fixation duration on the nose and mouth when noise was added.The increase in gaze fixations on the central part of the face is consistent with the fact that direct foveation of the mouth is not required in order to gather visual speech information (Paré et al, 2003). Rather crude video of the face, which has been degraded by either pixilation (MacDonald, Andersen & Bachmann, 2000) or by spatial frequency filtering (Munhall, Kroos, Jozan & Vatikiotis-Bateson, 2004), provides sufficient visual speech information to influence the perception of speech.…”

mentioning

confidence: 59%

“…There is no significant effect of talker variability on either the overall number of fixations or the median duration of fixations (p> .05). Detailed results for each region of interest (ROI) are presented below for the overall number of fixations and fixation durations, and for the number of first and second fixations in each trial.In previous studies looking at gaze behavior during face perception, specifically in studies of audiovisual speech perception (Everdell, Marsh, Yurick, Munhall & Paré, 2007; VatikiotisBateson et al, 1998;Paré et al, 2003) and identity judgment (Henderson et al, 2005;Barton et al, 2006;Althoff & Cohen, 1999), there is a preference to fixate the talker's right eye more often than the left eye. We wanted to see if this preference to fixate the right eye also occurred in the current study.…”

mentioning

confidence: 99%

“…This would provide an account for the considerable fixations made to the eyes during audiovisual speech experiments (e.g. Vatikiotis-Bateson, Eigsti, Yano & Munhall, 1998;Paré, Richler, ten Hove & Munhall, 2003;Lansing & McConkie, 2003) even though the eyes are not a particularly informative facial feature for perceiving speech sounds.Identity information is present in both the face and the voice and recently there has been considerable interest in the ability of both humans and animals to use this information (e.g., Ghazanfar et al, 2007). Individuals can estimate body size, age and gender from both faces and voices.…”

mentioning

confidence: 99%

“…To date, the active visual exploration of dynamic faces has only been investigated during audiovisual and visual-only speech perception (e.g. Vatikiotis-Bateson et al, 1998;Lansing & McConkie, 1999, 2003Paré et al, 2003;Buchan et al, 2007;Everdell et al, 2007) and so it is still unknown how an explicit identity task using dynamic facial stimuli might influence gaze behavior.A second, related possibility for the difference in mouth fixations is that the identity of a talking face is not restricted to the visual modality. Voices also contain information about identity and gaze fixations may be tuned to maximize the audiovisual integration of this information.…”

mentioning

confidence: 99%

See 3 more Smart Citations

The effect of varying talker identity and listening conditions on gaze behavior during audiovisual speech perception

2008

Self Cite

View full text Add to dashboard Cite

During face-to-face conversation the face provides auditory and visual linguistic information, and also conveys information about the identity of the speaker. This study investigated behavioral strategies involved in gathering visual information while watching talking faces. The effects of varying talker identity and varying the intelligibility of speech (by adding acoustic noise) on gaze behavior were measured with an eyetracker. Varying the intelligibility of the speech by adding noise had a noticeable effect on the location and duration of fixations. When noise was present subjects adopted a vantage point that was more centralized on the face by reducing the frequency of the fixations on the eyes and mouth and lengthening the duration of their gaze fixations on the nose and mouth. Varying talker identity resulted in a more modest change in gaze behavior that was modulated by the intelligibility of the speech. Although subjects generally used similar strategies to extract visual information in both talker variability conditions, when noise was absent there were more fixations on the mouth when viewing a different talker every trial as opposed to the same talker every trial. These findings provide a useful baseline for studies examining gaze behavior during audiovisual speech perception and perception of dynamic faces. IntroductionWe see and process faces every day in a wide variety of contexts, from line drawings of faces and static photographs, to dynamic movies and live faces during face-to-face communication. These faces contain a wealth of social, emotional, identity and linguistic information. Although a great deal of information can be gleaned from static faces, the motion of dynamic faces contains information about identity and emotion not present in static faces (Ambadar, Schooler & Cohn, 2005;Hill & Johnson, 2001;Knappmeyer, Thornton & Bülthoff, 2003;O'Toole, Roark, & Abdi, 2002;Lander & Bruce, 2000). Facial motion also contains linguistic information, as evidenced by the fact that silent speechreading is possible (Bernstein, Demorest & Tucker, 2000). Rarely though, is this visual speech information present in the complete absence of auditory speech information and audiovisual speech perception is the natural manner of communication. Visual speech information influences the perception of auditory speech in both perfectly audible conditions (McGurk & MacDonald, 1976, MacDonald & Correspondence: J.N. Buchan, Department of Psychology, Queen's University, Humphrey Hall, 62 Arch Street, Kingston, Ontario, Canada K7L 3N6, Tel: 613-533-6275, E-mail: 2jnb@queensu.ca. Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and al...

show abstract

mentioning

confidence: 59%

mentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

The effect of varying talker identity and listening conditions on gaze behavior during audiovisual speech perception

2008

Self Cite

View full text Add to dashboard Cite

show abstract

“…It is also interesting to see that the time spent looking at the mouth has lessened when gestures are present, this means that when gestures are generally present on the face, they draw the gaze away from the mouth. This, nonetheless, does not mean that the information extracted from the mouth region is reduced when gestures are synthesized since peripheral vision can still help perceiving lip movements even if the eyes are not fixated on the lips, as found in [38]. …”

Section: Gaze Analysismentioning

confidence: 94%

Audio-Visual Prosody: Perception, Detection, and Synthesis of Prominence

Moubayed

Beskow

Granström

et al. 2011

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. In this chapter, we investigate the effects of facial prominence cues, in terms of gestures, when synthesized on animated talking heads. In the first study a speech intelligibility experiment is conducted, where speech quality is acoustically degraded, then the speech is presented to 12 subjects through a lip synchronized talking head carrying head-nods and eyebrow raising gestures. The experiment shows that perceiving visual prominence as gestures, synchronized with the auditory prominence, significantly increases speech intelligibility compared to when these gestures are randomly added to speech. We also present a study examining the perception of the behavior of the talking heads when gestures are added at pitch movements. Using eyegaze tracking technology and questionnaires for 10 moderately hearing impaired subjects, the results of the gaze data show that users look at the face in a similar fashion to when they look at a natural face when gestures are coupled with pitch movements opposed to when the face carries no gestures. From the questionnaires, the results also show that these gestures significantly increase the naturalness and helpfulness of the talking head.

show abstract

“Look who's talking!” Gaze Patterns for Implicit and Explicit Audio‐Visual Speech Synchrony Detection in Children With High‐Functioning Autism

et al. 2015

View full text Add to dashboard Cite

Lay Abstract Background Conversation requires integration of information from faces and voices to fully understand the speaker’s message. To detect auditory-visual asynchrony of speech, listeners must integrate visual movements of the face, particularly the mouth, with auditory speech information. Individuals with autism spectrum disorder (ASD) may be less successful at such multisensory integration, despite their demonstrated preference for looking at the mouth region of a speaker. Method We showed a split-screen video of two identical individuals speaking side by side. Only one of the speakers was in synchrony with the corresponding audio track and synchrony switched between the two speakers every few seconds. Participants had to watch the video, without further instructions (implicit condition) or specifically watch the in-synch speaker (explicit condition). We recorded which part of the screen and face their eyes targeted Participants Individuals with and without high-functioning autism (HFA) aged 8–19. Results Both groups looked at the in-synch video significantly more with explicit instructions. However, participants with HFA looked at the in-synch video less than typically developing (TD) peers and did not increase their gaze time as much as TD participants in the explicit task. Importantly, the HFA group looked significantly less at the mouth than their TD peers, and significantly more at the non-face regions of the image. There were no between-group differences for eye-directed gaze. Conclusions Individuals with HFA spend less time looking at the crucially important mouth region of the face during auditory-visual speech integration, which is non-effective gaze behavior for this type of task. Scientific Abstract Background Conversation requires integration of information from faces and voices to fully understand the speaker’s message. To detect auditory-visual asynchrony of speech listeners must integrate visual movements of the face, particularly the mouth, with auditory speech information. Individuals with autism spectrum disorder (ASD) may be less successful at such multisensory integration, despite their demonstrated preference for looking at the mouth region of a speaker. Method We showed a split-screen video of two identical individuals speaking side by side. Only one of the speakers was in synchrony with the corresponding audio track and synchrony switched between the two speakers every few seconds. Participants were asked to watch the video, without further instructions (implicit condition) or to specifically watch the in-synch speaker (explicit condition). We recorded which part of the screen and face their eyes targeted. Participants Individuals with and without high-functioning autism (HFA) aged 8–19. Results Both groups looked at the in-synch video significantly more with explicit instructions. However, participants with HFA looked at the in-synch video less than typically developing (TD) peers and did not increase their gaze time as much as TD participants in the explicit...

show abstract

Gaze behavior in audiovisual speech perception: The influence of ocular fixations on the McGurk effect

Cited by 70 publications

References 51 publications

The effect of varying talker identity and listening conditions on gaze behavior during audiovisual speech perception

The effect of varying talker identity and listening conditions on gaze behavior during audiovisual speech perception

Audio-Visual Prosody: Perception, Detection, and Synthesis of Prominence

“Look who's talking!” Gaze Patterns for Implicit and Explicit Audio‐Visual Speech Synchrony Detection in Children With High‐Functioning Autism

Contact Info

Product

Resources

About