Assessing the effect of physical differences in the articulation of consonants and vowels on audiovisual temporal perception

Abstract

We investigated how the physical differences associated with the articulation of speech affect the temporal aspects of audiovisual speech perception. Video clips of consonants and vowels uttered by three different speakers were presented. The video clips were analyzed using an auditory-visual signal saliency model in order to compare signal saliency and behavioral data. Participants made temporal order judgments (TOJs) regarding which speech-stream (auditory or visual) had been presented first. The sensitivity of participants' TOJs and the point of subjective simultaneity (PSS) were analyzed as a function of the place, manner of articulation, and voicing for consonants, and the height/backness of the tongue and lip-roundedness for vowels. We expected that in the case of the place of articulation and roundedness, where the visual-speech signal is more salient, temporal perception of speech would be modulated by the visual-speech signal. No such effect was expected for the manner of articulation or height. The results demonstrate that for place and manner of articulation, participants' temporal percept was affected (although not always significantly) by highly-salient speech-signals with the visual-signals requiring smaller visual-leads at the PSS. This was not the case when height was evaluated. These findings suggest that in the case of audiovisual speech perception, a highly salient visual-speech signal may lead to higher probabilities regarding the identity of the auditory-signal that modulate the temporal window of multisensory integration of the speech-stimulus.

The unity assumption facilitates cross-modal binding of musical, non-speech stimuli: The role of spectral and amplitude envelope cues

“…However, the inferred causal source of an auditory event should not be important in determining the PSS, and likely depends, rather, on a number of low level stimulus factors. Indeed, the wide variability in PSS values across experiments in the literature suggests that a number of low-level factors might affect this measure, with considerable disagreement on both the size and direction of the temporal asymmetry window (Vatakis, Maragos, Rodomagoulakis, & Spence, 2012). Some studies with audiovisual speech stimuli have found that simultaneity is perceived when the visual stream leads (Dixon & Spitz, 1980;Grant, Van Wassenhove, & Poeppel, 2004), while other research has found that the modality lead/lag differs depending on whether a phoneme or syllable is presented (Vatakis & Spence, 2006a).…”

Assessing the role of attention in the audiovisual integration of speech

“…That is, the lips usually move before any sounds are produced. This means that, for certain phonemes, the visual speech signal comes sometimes far ahead, and can even predict, the auditory speech sound [24,74,77,80].…”

Tolerance for audiovisual asynchrony is enhanced by the spectrotemporal fidelity of the speaker’s mouth movements and speech

“…However, the link between the information conveyed by mouth movements and the unfolding speech signal is not a simple relationship. This association varies with the type of phoneme and with the complexity of the acoustics and information conveyed by mouth movements (Cappelletta & Harte, 2012; Jesse & Massaro, 2010; Vatakis, Maragos, Rodomagoulakis, & Spence, 2012; Vatakis & Spence, 2006). For example, in visual speech, certain phonemes are more visible than others (e.g., /b/ vs. /k/), and even visible phonemes are not always unique, as /b/ cannot be distinguished from /m/ or /p/ by speech-reading alone.…”