Effects of Phonetic Context on Audio-Visual Intelligibility of French

Abstract

Bimodal perception leads to better speech understanding than auditory perception alone. We evaluated the overall benefit of lip-reading on natural utterances of French produced by a single speaker. Eighteen French subjects with good audition and vision were administered a closed set identification test of VCVCV nonsense words consisting of three vowels [i, a, y] and six consonants [b, v, z, 3, r , l]. Stimuli were presented under both auditory and audio-visual conditions with white noise added at various signal-to-noise ratios. Identification scores were higher in the bimodal condition than in the auditory-alone condition, especially in situations where acoustic information was reduced. The auditory and audio-visual intelligibility of the three vowels [i, a, y] averaged over the six consonantal contexts was evaluated as well. Two different hierarchies of intelligibility were found. Auditorily, [a] was most intelligible, followed by [i] and then by [y]; whereas visually [y] was most intelligible, followed by [a] and [i]. We also quantified the contextual effects of the three vowels on the auditory and audio-visual intelligibility of the consonants. Both the auditory and the audio-visual intelligibility of surrounding consonants was highest in the [a] context, followed by the [i] context and lastly the [y] context.

Perceptual Fusion and Stimulus Coincidence in the Cross-Modal Integration of Speech

“…Merging information from different senses confers distinct behavioral advantages, enabling faster and more accurate discrimination than with unimodal stimuli (Hershenson, 1962;Morrell, 1968;Stein et al, 1989;Perrott et al, 1990;Hughes et al, 1994;Frens et al, 1995), especially when the signals are degraded (Sumby and Pollack, 1954;MacLeod and Summerfield, 1987;Perrott et al, 1991;Benoit et al, 1994). To realize these advantages, the brain continually coordinates sensory inputs across the audiovisual (Calvert et al, 2000;Grant and Seitz, 2000;Shams et al, 2002;Callan et al, 2003), visual-tactile (Banati et al, 2000;Macaluso et al, 2000;Stein et al, 2001), and audiosomatic (Schulz et al, 2003) domains and combines them into coherent perceptions.…”

Recognizing Prosody from the Lips

“…This is consistent with the finding that lip protrusion is the most visible lip feature (Benoît et al, 1994).…”

Adaptive Decision Fusion for Audio-Visual Speech Recognition

“…Second, the two modalities produce "synergy. ": Performance of audio-visual speech perception can outperform those of acoustic-only and visual-only perception for diverse noise conditions (Benoît et al, 1994).…”