Multimodal perceptual organization of speech: Evidence from tone analogs of spoken utterances

Abstract

Theoretical and practical motives alike have prompted recent investigations of multimodal speech perception. Theoretically, multimodal studies have extended the conceptualization of perceptual organization beyond the familiar modality-bound accounts deriving from Gestalt psychology. Practically, such investigations have been driven by a need to understand the proficiency of multimodal speech perception using an electrocochlear prosthesis for hearing. In each domain, studies have shown that perceptual organization of speech can occur even when the perceiver's auditory experience departs from natural speech qualities. Accordingly, our research examined auditor-visual multimodal integration of videotaped faces and selected acoustic constituents of speech signals, each realized as a single sinewave tone accompanying a video image of an articulating face. The single tone reproduced the frequency and amplitude of the phonatory cycle or of one of the lower three oral formants. Our results showed a distinct advantage for the condition pairing the video image of the face with a sinewave replicating the second formant, despite its unnatural timbre and its presentation in acoustic isolation from the rest of the speech signal. Perceptual coherence of multimodal speech in these circumstances is established when the two modalities concurrently specify the same underlying phonetic attributes.

Visibility of speech articulation enhances auditory phonetic convergence

“…However, there is plenty of precedence to suggest that our observed effects would generalize to other diverse samples of models and shadowers of both genders. Within the speech literature, robust visual (for reviews, see Fowler, 2004;Rosenblum, 2008) and lexical (for a review, see Luce & McLennan, 2005) influences on auditory speech perception are reported across diverse samples of models and participants, with many studies employing stimuli derived from a single model talker (e.g., Brancazio, 2004;Erber, 1971;Luce, Pisoni, & Goldinger, 1990;Remez et al, 1998;Rosenblum et al, 1996;Ross et al, 2007;Tye-Murray, Sommers, & Spehar, 2007). Within the phonetic convergence literature (much of which is cited in this article), automatic imitation of perceived speech has been observed across studies employing diverse samples of models, shadowers, and conversational partners.…”

“…Perceivers can also demonstrate an enhanced ability to identify degraded auditory speech when presented with the concurrent visual speech component of the auditory stimulus. This visual enhancement of auditory speech perception has been demonstrated when identifying speech presented in noise (e.g., Erber, 1975;Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007;Sumby & Pollack, 1954) and acoustically transformed speech (e.g., Remez, Fellowes, Pisoni, Goh, & Rubin, 1998). The visibility of a speaker can also improve the comprehension of accented speech (e.g., Sueyoshi & Hardison, 2005) and speech that conveys complicated content (e.g., Arnold & Hill, 2001;Reisberg, McLean, & Goldfield, 1987).…”

Talker and Lexical Effects on Audiovisual Word Recognition by Adults With Cochlear Implants

“…The relations between the two modalities, then, are specified by the information relating each pattern to the common, underlying, dynamic vocal tract gestures of the talker that produced them. It is precisely this time-varying articulatory behavior of the vocal tract that has been shown to be of primary importance in the perception of speech (Liberman, Cooper, Shankweiler, & Studdert-Kennedy, 1967;Remez, Fellowes, Pisoni, Goh, & Rubin, 1998;Remez, Rubin, Pisoni, & Carrell, 1981;Remez, Rubin, Berns, Pardo, & Lang, 1994).…”

“…With this conceptualization in mind, the sensory and perceptual information relevant for speech perception is modality-neutral or amodal, because it can be carried by more than one sensory modality (Fowler, 1986; Gaver, 1993;Remez et al, 1998;Rosenblum & Saldaña, 1996). The amodal nature of phonetic information is demonstrated convincingly in studies showing that perceptual information obtained via the tactile modality, in the form of Tadoma, can be used and integrated across sensory modalities in speech perception (Fowler & Dekle, 1991), albeit with limited utility.…”

“…The relations between the two modalities, then, are specified by the information relating each pattern to the common, underlying, dynamic vocal tract gestures of the talker that produced them. It is precisely this time-varying articulatory behavior of the vocal tract that has been shown to be of primary importance in the perception of speech (Liberman, Cooper, Shankweiler, & Studdert-Kennedy, 1967;Remez, Fellowes, Pisoni, Goh, & Rubin, 1998;Remez, Rubin, Pisoni, & Carrell, 1981;Remez, Rubin, Berns, Pardo, & Lang, 1994).With this conceptualization in mind, the sensory and perceptual information relevant for speech perception is modality-neutral or amodal, because it can be carried by more than one sensory modality (Fowler, 1986; Gaver, 1993;Remez et al, 1998;Rosenblum & Saldaña, 1996). The amodal nature of phonetic information is demonstrated convincingly in studies showing that perceptual information obtained via the tactile modality, in the form of Tadoma, can be used and integrated across sensory modalities in speech perception (Fowler & Dekle, 1991), albeit with limited utility.…”