Animated virtual characters to explore audio-visual speech in controlled and naturalistic environments

Abstract

Natural speech is processed in the brain as a mixture of auditory and visual features. An example of the importance of visual speech is the McGurk effect and related perceptual illusions that result from mismatching auditory and visual syllables. Although the McGurk effect has widely been applied to the exploration of audio-visual speech processing, it relies on isolated syllables, which severely limits the conclusions that can be drawn from the paradigm. In addition, the extreme variability and the quality of the stimuli usually employed prevents comparability across studies. To overcome these limitations, we present an innovative methodology using 3D virtual characters with realistic lip movements synchronized on computer-synthesized speech. We used commercially accessible and affordable tools to facilitate reproducibility and comparability, and the set-up was validated on 24 participants performing a perception task. Within complete and meaningful French sentences, we paired a labiodental fricative viseme (i.e. /v/) with a bilabial occlusive phoneme (i.e. /b/). This audiovisual mismatch is known to induce the illusion of hearing /v/ in a proportion of trials. We tested the rate of the illusion while varying the magnitude of background noise and audiovisual lag. Overall, the effect was observed in 40% of trials. The proportion rose to about 50% with added background noise and up to 66% when controlling for phonetic features. Our results conclusively demonstrate that computer-generated speech stimuli are judicious, and that they can supplement natural speech with higher control over stimulus timing and content.

The phase of cortical oscillations determines the perceptual fate of visual cues in naturalistic audiovisual speech

“…We designed audiovisual speech stimuli by combining speechsynthesized sentences with virtual characters whose lip movements were animated synchronously to the speech sounds (20) (Fig. 1A).…”

“…The observed values were expressed as a z score relative to that distribution, and a two-sided P value was computed from the z score. At each electrode, frequency, and time point, P values were then combined over participants using Stouffer's method (20). The resulting P values express the probability that the observed values be observed if all null hypotheses are true.…”

The McGurk Effect Across Languages

“…Though the effect was conceived by the mismatch of sound and video, not all speech signals perceived in our day-to-day conversations happen through a two-dimensional representation of our three-dimensional selves (as ubiquitous as online videoconferencing is becoming in today's world). Future research could further examine the McGurk effect in a 3D space using virtual reality technology in more phonologically complex languages like those mentioned in this review, building on the studies that have already been done in the more phonologically simple English and French languages (Siddig, Sun, Sun, Parker & Hines, 2019;Thézé et al, 2020). Another limitation is the use of monophonic or stereophonic playback (using one or two audio channels) through speakers or headphones, which makes it difficult to mimic true ambient sound as it is easier to localize the origin of the signal, whereas in a noisy environment, the disruptive signals that confuse a listener can be projected from any direction (Snow, 1953;Yao et al, 2020).…”

The Effect of Using Web-Based Interactive Learning Media for Vocational High School Students to Understanding of Looping: Qualitative Approach

“…Audio explanations of the material accompany the learning videos. Using audio-video in learning can encourage students' active participation in their hearing and vision, causing them to become more focused on their studies (Thézé et al, 2020).…”