The development of gaze to a speaking face

Irwin, Julia; Brancazio, Lawrence; Volpe, Nicole

doi:10.1121/1.4982727

Cited by 4 publications

(7 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In terms of accuracy data (measured by a button press response within the active experiment), participants demonstrated lower accuracy levels within the AV condition compared to the PX condition, suggesting that participants did experience a phonemic restoration effect when the presence of articulatory information from the mouth was available. This result is consistent with our previous studies on phonemic restoration in adults and children ( Irwin et al, 2017a , b , 2022 ). Identification of the standard stimuli within the active experiment were significantly better in the AV condition compared to the PX condition.…”

Section: Discussionsupporting

confidence: 94%

“…For many years, there has been evidence that visual information about speech influences what listeners hear, including increasing identification of the speech signal in the context of background noise ( Sumby and Pollack, 1954 ; Grant et al, 1998 ). This influence of visual speech has been reported for a wide range of ages ( Irwin et al, 2017a , b ) for persons’ with typical and reduced hearing ( Sommers et al, 2005 ), for clinical populations such as for persons with autism ( Kuhl et al, 2005 ; Stevenson et al, 2014 ; Irwin et al, 2022 ), and for nonnative speakers of English ( Reisberg et al, 1987 ). The presence of visual articulatory information can also facilitate the perception of heard speech, speeding up cortical processing of the auditory signal ( van Wassenhove et al, 2005 ) and facilitating language processing ( MacDonald et al, 2000 ; Lachs and Pisoni, 2004 ).…”

Section: Introductionmentioning

confidence: 79%

“…The stimuli used in this study are identical to those in Irwin et al (2017a , b) . Stimuli were created by videotaping and recording an adult male speaker of English producing the syllable /ba/.…”

Section: Methodsmentioning

confidence: 99%

“…Video of the speaker’s face does not change (always producing /ba/), but the auditory stimuli (/ba/ or / a /) vary. Thus, when the / a / auditory stimulus is dubbed over the visual /ba/, a visual influence will effectively “restore” the weakened auditory cues so that the stimulus is perceived as a /ba/, akin to a visual phonemic restoration effect ( Samuel, 1981 ; Kashino, 2006 ; Irwin et al, 2011 , 2017a , b , 2022 ; Irwin and Brancazio, 2014 ).…”

Section: Methodsmentioning

confidence: 99%

“…Research on audiovisual communication has extended into child populations with similar results. Irwin et al (2017a , b) demonstrated that typically-developing children between the ages of 5–10 years show increased gaze to the mouth of a speaker in a range of audiovisual environments, including audiovisual speech, audiovisual speech in noise, and in an audiovisual mismatch condition.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Where on the face do we look during phonemic restoration: An eye-tracking study

Baron,

Harwood,

Kleinman

et al. 2023

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

Face to face communication typically involves audio and visual components to the speech signal. To examine the effect of task demands on gaze patterns in response to a speaking face, adults participated in two eye-tracking experiments with an audiovisual (articulatory information from the mouth was visible) and a pixelated condition (articulatory information was not visible). Further, task demands were manipulated by having listeners respond in a passive (no response) or an active (button press response) context. The active experiment required participants to discriminate between speech stimuli and was designed to mimic environmental situations which require one to use visual information to disambiguate the speaker’s message, simulating different listening conditions in real-world settings. Stimuli included a clear exemplar of the syllable /ba/ and a second exemplar in which the formant initial consonant was reduced creating an /a/−like consonant. Consistent with our hypothesis, results revealed that the greatest fixations to the mouth were present in the audiovisual active experiment and visual articulatory information led to a phonemic restoration effect for the /a/ speech token. In the pixelated condition, participants fixated on the eyes, and discrimination of the deviant token within the active experiment was significantly greater than the audiovisual condition. These results suggest that when required to disambiguate changes in speech, adults may look to the mouth for additional cues to support processing when it is available.

show abstract

Section: Discussionsupporting

confidence: 94%

Section: Introductionmentioning

confidence: 79%

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Where on the face do we look during phonemic restoration: An eye-tracking study

Baron,

Harwood,

Kleinman

et al. 2023

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

show abstract

Psychobiological Responses Reveal Audiovisual Noise Differentially Challenges Speech Recognition

et al. 2019

View full text Add to dashboard Cite

Objectives: In noisy environments, listeners benefit from both hearing and seeing a talker, demonstrating audiovisual (AV) cues enhance speech-in-noise (SIN) recognition. Here, we examined the relative contribution of auditory and visual cues to SIN perception and the strategies used by listeners to decipher speech in noise interference(s). Design: Normal-hearing listeners (n = 22) performed an open-set speech recognition task while viewing audiovisual TIMIT sentences presented under different combinations of signal degradation including visual (AVn), audio (AnV), or multimodal (AnVn) noise. Acoustic and visual noises were matched in physical signal-to-noise ratio. Eyetracking monitored participants’ gaze to different parts of a talker’s face during SIN perception. Results: As expected, behavioral performance for clean sentence recognition was better for A-only and AV compared to V-only speech. Similarly, with noise in the auditory channel (AnV and AnVn speech), performance was aided by the addition of visual cues of the talker regardless of whether the visual channel contained noise, confirming a multimodal benefit to SIN recognition. The addition of visual noise (AVn) obscuring the talker’s face had little effect on speech recognition by itself. Listeners’ eye gaze fixations were biased toward the eyes (decreased at the mouth) whenever the auditory channel was compromised. Fixating on the eyes was negatively associated with SIN recognition performance. Eye gazes on the mouth versus eyes of the face also depended on the gender of the talker. Conclusions: Collectively, results suggest listeners (1) depend heavily on the auditory over visual channel when seeing and hearing speech and (2) alter their visual strategy from viewing the mouth to viewing the eyes of a talker with signal degradations, which negatively affects speech perception.

show abstract