Language identification from visual-only speech signals

Abstract

Our goal in the present study was to examine how observers identify English and Spanish from visual-only displays of speech. First, we replicated the recent findings of Soto-Faraco et al. (2007) with Spanish and English bilingual and monolingual observers using different languages and a different experimental paradigm (identification). We found that prior linguistic experience affected response bias but not sensitivity (Experiment 1). In two additional experiments, we investigated the visual cues that observers use to complete the language-identification task. The results of Experiment 2 indicate that some lexical information is available in the visual signal but that it is limited. Acoustic analyses confirmed that our Spanish and English stimuli differed acoustically with respect to linguistic rhythmic categories. In Experiment 3, we tested whether this rhythmic difference could be used by observers to identify the language when the visual stimuli is temporally reversed, thereby eliminating lexical information but retaining rhythmic differences. The participants performed above chance even in the backward condition, suggesting that the rhythmic differences between the two languages may aid language identification in visual-only speech signals. The results of Experiments 3A and 3B also confirm previous findings that increased stimulus length facilitates language identification. Taken together, the results of these three experiments replicate earlier findings and also show that prior linguistic experience, lexical information, rhythmic structure, and utterance length influence visual-only language identification.A large body of research has demonstrated that speech perception is multimodal in nature. In addition to the auditory properties of speech, the visual signal carries important information about the phonetic structure of the message that affects the perception and comprehension of the speech signal (see, e.g., Massaro, 1987;Sumby & Pollack, 1954;Summerfield, 1987). The visual aspects of speech have been shown to both enhance and alter the perception of the auditory speech signal for listeners with hearing impairment, as well as for normalhearing listeners (see, e.g., R. Campbell & Dodd, 1980;Hamilton, Shenton, & Coslett, 2006;Kaiser, Kirk, Lachs, & Pisoni, 2003;Lachs, 1999;Lachs, Weiss, & Pisoni, 2002;Summerfield, 1987).In their seminal study of audio-visual speech perception, Sumby and Pollack (1954) showed that the addition of visual information dramatically improved speech intelligibility at less favorable signal-to-noise ratios in normal-hearing listeners. When presented with degraded auditory signals, the observers experienced large gains in intelligibility of speech signals in the auditory-visual conditions relative to the auditory-only conditions. The contribution of Correspondence concerning this article should be addressed to R. E. Ronquest, Speech Research Laboratory, Department of Psychological and Brain Sciences, Indiana University, 1101 East Tenth Street, Bloomington, IN 47405-1301 (rr...

Language Identification Using Visual Features

“…Another feature of language is rhythm. [41] explains that babies have the ability to distinguish languages based on acoustic rhythm, and [42] suggests that adults also have this ability and furthermore, rhythm is expressed visually. Further work into VLID could therefore focus on incorporating both of these additional language cues and evaluating their contribution to language discrimination.…”

Discrimination Between Native and Non-Native Speech Using Visual Features Only

“…Specifically, Ronquest et al [21] carry out experiments in which participants are asked to observe, without listening, video clips that show a male or a female speaker talking in English or Spanish. Both speakers appearing in the videos are bilingual in English and Spanish.…”

“…However, the beneficial role of visual information to speech comprehension has been well documented [17] and experimentally validated [18]- [20]. Furthermore, recent findings indicate that human observers can actually perform language identification through the visual modality only [21]. Automated approaches for visual-only language identification have also been proposed (e.g., [22]).…”

Rhythm on Your Lips

“…For example, it has been found that both bilingual and monolingual Spanish and Catalan speakers, but not speakers of English and Italian, can discriminate Catalan and Spanish using only visual cues (Soto-Faraco et al, 2007). Also both monolingual and bilingual English- and Spanish-speaking adults have been shown to discriminate between Spanish and English only on the basis visual cues (Ronquest et al, 2010). Likewise, Navarra et al (2014) showed that English and Spanish/Catalan adult speakers do exploit visual information concerning the temporal distribution of consonant and vowel intervals to discriminate languages that differ in this speech property such as English and Japanese.…”

“…For example, bilingual Spanish-Catalan as well as monolingual Spanish and Catalan speakers, but not monolingual speakers of English and Italian, can discriminate Catalan and Spanish using only visual cues (Soto-Faraco et al, 2007). Also monolingual and bilingual English- and Spanish-speaking adults have been shown to discriminate between Spanish and English—two languages differing at the basic rhythmic level—only on the basis of the visual cues provided by speaking faces (Ronquest et al, 2010). These results suggest that adult listeners can discriminate between rhythmically similar (Spanish and Catalan) as well as rhythmically different (English and Spanish) languages by analyzing the facial mimic when they know at least one of the two languages.…”