The perception of speaking rate using visual information from a talker’s face

Green, Kerry P.

doi:10.3758/bf03207990

Cited by 19 publications

(9 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Green (1987) reported that the subjects' ratings of speaking rate in auditory, visual, and audiovisual presentations did not differ, and our results are consistent with this finding. Although in the present data there seems to have been a greater range of effects due to visual speaking condition, the pattern of change was the same for both modalities.…”

Section: Discussionsupporting

confidence: 92%

See 1 more Smart Citation

Temporal constraints on the McGurk effect

Munhall

Gribble

Sacco

et al. 1996

Perception & Psychophysics

276

267

View full text Add to dashboard Cite

Three experiments are reported on the influence of different timing relations on the McGurk effect. In the first experiment, it is shown that strict temporal synchrony between auditory and visual speech stimuli is not required for the McGurk effect. Subjects were strongly influenced by the visual stimuli when the auditory stimuli lagged the visual stimuli by as much as 180 msec. In addition, a stronger McGurk effect was found when the visual and auditory vowels matched. In the second experiment, we paired auditory and visual speech stimuli produced under different speaking conditions (fast, normal, clear). The results showed that the manipulations in both the visual and auditory speaking conditions independently influenced perception. In addition, there was a small but reliable tendency for the better matched stimuli to elicit more McGurk responses than unmatched conditions. In the third experiment, we combined auditory and visual stimuli produced under different speaking conditions (fast, clear) and delayed the acoustics with respect to the visual stimuli. The subjects showed the same pattern of results as in the second experiment. Finally, the delay did not cause different patterns of results for the different audiovisual speaking style combinations. The results suggest that perceivers may be sensitive to the concordance of the time-varying aspects of speech but they do not require temporal coincidence of that information.When the face moves during speech production it provides information about the place of articulation as well as the class of phoneme that is produced. Evidence from studies of lipreading as well as studies of speech in noise (e.g., Sumby & Pollack, 1954) suggest that perceivers can gain significant amounts of information about the speech target through the visual channel. How this information is combined with speech acoustics to form a single percept, however, is not clear. One useful approach to studying audiovisual integration in speech is to dub various auditory stimuli onto different visual speech stimuli. When a discrepancy exists between the information from the two modalities, subjects fuse the visual and auditory information to form a new percept. For example, when the face articulates /gi/ and the auditory stimulus is /bi/, many subjects report hearing /di/.

show abstract

Section: Discussionsupporting

confidence: 92%

“…We cannot determine the source of this difference from the present data. As Green (1987) suggested, however, it may simply be that speaking rate and the type of rate measured by Welch et al differed in terms of their auditory dominance.…”

Section: Discussionmentioning

confidence: 93%

Temporal constraints on the McGurk effect

Munhall

Gribble

Sacco

et al. 1996

Perception & Psychophysics

276

267

View full text Add to dashboard Cite

show abstract

“…However, because the visemes associated with vowels and consonants are retained in the backward stimuli and because temporal reversal does not alter the duration of these visemes, the temporal/rhythmic differences between Spanish and English should be relatively well maintained under this manipulation. Furthermore, previous studies on visual-only speech perception have reported that observers are able to extract temporally dependent information such as speaking-rate and stress differences from visual-only displays of speech (Bernstein, Eberhardt, & Demorest, 1986; Green, 1987). Thus, backward versions of the visual-only stimuli were included in the stimulus set to assess whether the participants could make accurate judgments about the language when lexical information was unavailable.…”

Section: Experiments 3a and 3b Rhythmic Cues To Language Identificationmentioning

confidence: 99%

Language identification from visual-only speech signals

Ronquest

Levi

Pisoni

2010

Attention, Perception, & Psychophysics

View full text Add to dashboard Cite

Our goal in the present study was to examine how observers identify English and Spanish from visual-only displays of speech. First, we replicated the recent findings of Soto-Faraco et al. (2007) with Spanish and English bilingual and monolingual observers using different languages and a different experimental paradigm (identification). We found that prior linguistic experience affected response bias but not sensitivity (Experiment 1). In two additional experiments, we investigated the visual cues that observers use to complete the language-identification task. The results of Experiment 2 indicate that some lexical information is available in the visual signal but that it is limited. Acoustic analyses confirmed that our Spanish and English stimuli differed acoustically with respect to linguistic rhythmic categories. In Experiment 3, we tested whether this rhythmic difference could be used by observers to identify the language when the visual stimuli is temporally reversed, thereby eliminating lexical information but retaining rhythmic differences. The participants performed above chance even in the backward condition, suggesting that the rhythmic differences between the two languages may aid language identification in visual-only speech signals. The results of Experiments 3A and 3B also confirm previous findings that increased stimulus length facilitates language identification. Taken together, the results of these three experiments replicate earlier findings and also show that prior linguistic experience, lexical information, rhythmic structure, and utterance length influence visual-only language identification.A large body of research has demonstrated that speech perception is multimodal in nature. In addition to the auditory properties of speech, the visual signal carries important information about the phonetic structure of the message that affects the perception and comprehension of the speech signal (see, e.g., Massaro, 1987;Sumby & Pollack, 1954;Summerfield, 1987). The visual aspects of speech have been shown to both enhance and alter the perception of the auditory speech signal for listeners with hearing impairment, as well as for normalhearing listeners (see, e.g., R. Campbell & Dodd, 1980;Hamilton, Shenton, & Coslett, 2006;Kaiser, Kirk, Lachs, & Pisoni, 2003;Lachs, 1999;Lachs, Weiss, & Pisoni, 2002;Summerfield, 1987).In their seminal study of audio-visual speech perception, Sumby and Pollack (1954) showed that the addition of visual information dramatically improved speech intelligibility at less favorable signal-to-noise ratios in normal-hearing listeners. When presented with degraded auditory signals, the observers experienced large gains in intelligibility of speech signals in the auditory-visual conditions relative to the auditory-only conditions. The contribution of Correspondence concerning this article should be addressed to R. E. Ronquest, Speech Research Laboratory, Department of Psychological and Brain Sciences, Indiana University, 1101 East Tenth Street, Bloomington, IN 47405-1301 (rr...

show abstract

“…For infants, visuofacial information is commandeered to aid speech acquisition (Kent, 1984). For both children and adults, watching concordant articulatory gestures improves the perception of artificially degraded speech (e.g., Grant, 2001) and speech in noise (MacLeod and Summerfield, 1987;Green, 1987;Middelweerd and Plomp, 1987). Observing facial movements that do not match the acoustic speech can drastically change what people "hear", even when the acoustic signal is clear.…”

mentioning

confidence: 99%

Seeing speech affects acoustic information processing in the human brainstem

et al. 2005

View full text Add to dashboard Cite

Afferent auditory processing in the human brainstem is generally assumed to be determined by acoustic stimulus features and immune to stimulation by other senses or cognitive factors. In contrast, we show that lipreading during speech perception influences acoustic processing astonishingly early. Event-related brainstem potentials were recorded from 10 healthy adults to concordant (acousticvisual match), conflicting (acoustic-visual mismatch) and unimodal stimuli. Audiovisual interactions occurred around 11ms post-stimulation and persisted for the first 30ms of the response. Furthermore, response timing and magnitude depended on audiovisual pairings. These findings indicate that early auditory processing is more plastic than previously thought. Keywordsauditory; brainstem; multisensory; visual; speech Natural perceptions are rich with sensations from the auditory and visual modalities (Marks, 1982). As a friend says hello, we are cheered by their friendly tone and the sight of their smile. At a concert, we are amazed at the sight and sound of a trumpet player's technique. Real-world audiovisual experiences do not evoke distinct "multisensory" sensations. Although the combination of acoustic and visual information goes seamlessly unnoticed by the perceiver, it has a strikingly potent effect on perception (Marks, 2004). One of the most prevalent models of audiovisual integration posits that information from different modalities is processed in a hierarchical fashion along unisensory streams, which converge in higher-order structures (e.g., Massaro, 1998). The combined representation is then processed in a feed-forward fashion that does not affect doenstream processing. While this hypothesis has proven to account for copious multisensory phenomena, evidence of audiovisual interaction in lower-order structures encourages modification of the model. These observations have prompted a growing number of scientists to investigate the underlying neural mechanisms of audiovisual integration, or how, where and when in the brain we bind one sensation to another. *Corresponding author Gabriella A.E. Musacchia, Email: g-musacchia@northwestern.edu, Address: Department of Communication Sciences; Northwestern University, 2240 Campus Dr., Evanston, IL 60208, USA, Tel: 847.491.2465, Fax: 847.491 Many phenomena have shown that visual information has a remarkable influence on acoustic perception (Spence and McDonald, 2004). When acoustic and visual cues occur in close temporal and spatial proximity, their combined information enhances orientation (Zambarbieri, 2002), detection , classification (Ben Artzi and Marks, 1995), and reaction time (McDonald and Ward, 2000). Circumstances in which the acoustic and visual stimuli are not matched can cause a variety of illusions including ventriloquism (Howard and Templeton, 1966), and fused perceptions that do not correspond to either unimodal stimulus (McGurk and MacDonald, 1976;Massaro, 1998).One of the most ubiquitous and well-studied examples of audiovisual integration in humans is...

show abstract

The perception of speaking rate using visual information from a talker’s face

Cited by 19 publications

References 35 publications

Temporal constraints on the McGurk effect

Temporal constraints on the McGurk effect

Language identification from visual-only speech signals

Seeing speech affects acoustic information processing in the human brainstem

Contact Info

Product

Resources

About