Cross-linguistic perception of clearly spoken English tense and lax vowels based on auditory, visual, and auditory-visual information

Redmon, Charles; Leung, Keith K. W.; Wang, Yue; McMurray, Bob; Jongman, Allard; Sereno, Joan A.

doi:10.1016/j.wocn.2020.100980

Cited by 11 publications

(28 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Eighteen native English talkers (10 females, 8 males), born and raised in Western Canada, produced the target words both as plain (conversational) and as clear speech, where the speech styles were elicited using a simulated interactive computer speech recognition program established previously (Maniwa et al, 2009 ; Redmon et al, 2020 ; Tang et al, 2015 ). In this program, a talker was instructed to produce a target word naturally, as if in casual conversation (thus eliciting plain style productions).…”

Section: Methodsmentioning

confidence: 99%

“…Naturally produced clear speech has been shown to be as much as 7–38% more intelligible than its plain counterpart (Ferguson & Kewley-Port, 2002 ; Ferguson & Quené, 2014 ; Maniwa et al, 2008 ; Payton et al, 1994 ; Redmon et al, 2020 ; Uchanski et al, 1996 ). Clear speech can also improve intelligibility in visual (facial) speech perception (Gagné et al, 1994 , 2002 ; Helfer, 1997 ; Lander & Capek, 2013 ; Van Engen et al, 2014 ).…”

Section: Introductionmentioning

confidence: 99%

“…The results show that clear compared to plain speech productions involve longer and greater vertical lip stretch and jaw displacement across vowels, greater horizontal lip stretch for front unrounded vowels, and greater degree of lip rounding for rounded vowels. Further intelligibility research (Redmon et al, 2020 ), however, reveals that these visual cues are predictive of a clear-speech advantage for tense vowels and not lax vowels, presumably because lip stretching in clear speech deviates from lax vowel features and thus decreases tense-lax vowel distinctions (Leung et al, 2016 ). Indeed, clear-speech modifications are claimed to involve a trade-off between “contrast enhancement” and “maintenance of phonemic norms” in intelligibility benefit (Moon & Lindblom, 1994 ; Ohala, 1995 ; Smiljanić et al, 2021 ).…”

Section: Introductionmentioning

confidence: 99%

“…Indeed, clear-speech modifications are claimed to involve a trade-off between “contrast enhancement” and “maintenance of phonemic norms” in intelligibility benefit (Moon & Lindblom, 1994 ; Ohala, 1995 ; Smiljanić et al, 2021 ). Excessive exaggerations or modifications incompatible with sound-inherent cues (e.g., greater horizontal lip-stretching of lax vowel /ɪ/) may obscure visual distinctiveness between sounds (e.g., /i/-/ɪ/) and inhibit intelligibility (Redmon et al, 2020 ). These patterns indicate that clear-speech modifications need to be within category boundaries in order to benefit speech intelligibility.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Plain-to-clear speech video conversion for enhanced intelligibility

Sachdeva

Ruan

Hamarneh

et al. 2023

Int J Speech Technol

Self Cite

View full text Add to dashboard Cite

Clearly articulated speech, relative to plain-style speech, has been shown to improve intelligibility. We examine if visible speech cues in video only can be systematically modified to enhance clear-speech visual features and improve intelligibility. We extract clear-speech visual features of English words varying in vowels produced by multiple male and female talkers. Via a frame-by-frame image-warping based video generation method with a controllable parameter (displacement factor), we apply the extracted clear-speech visual features to videos of plain speech to synthesize clear speech videos. We evaluate the generated videos using a robust, state of the art AI Lip Reader as well as human intelligibility testing. The contributions of this study are: (1) we successfully extract relevant visual cues for video modifications across speech styles, and have achieved enhanced intelligibility for AI; (2) this work suggests that universal talker-independent clear-speech features may be utilized to modify any talker’s visual speech style; (3) we introduce “displacement factor” as a way of systematically scaling the magnitude of displacement modifications between speech styles; and (4) the high definition generated videos make them ideal candidates for human-centric intelligibility and perceptual training studies.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Plain-to-clear speech video conversion for enhanced intelligibility

Sachdeva

Ruan

Hamarneh

et al. 2023

Int J Speech Technol

Self Cite

View full text Add to dashboard Cite

show abstract

“…Platform under the Network Environment e establishment of an effective oral English teaching platform is the premise and basis for the transformation of effective teaching methods, and different modules are the key to operation of the design platform [10]. e design of effective oral English teaching platform mainly includes oral English perception processing module, oral English pronunciation acquisition program module, oral English pronunciation preprocessing module, and oral English pronunciation error detection modeling module.…”

Section: Effective Oral English Teachingmentioning

confidence: 99%

A Transformation of Oral English Teaching Method in the Network Environment

Yan-wei¹

2022

Scientific Programming

View full text Add to dashboard Cite

In order to effectually improve the application effect of the existing oral English teaching methods, this paper studies the conversion of effective oral English teaching methods under the network environment, aiming at the application problems, such as the flexibility of the current education platform. We build an effective oral English teaching platform which is comprised of oral English perception processing module, oral English pronunciation acquisition program module, oral English pronunciation preprocessing module, and oral English pronunciation error detection modeling module. We establish the information technology teaching objectives of the effective oral English teaching model on this basis. This research work is focused on the study of the output driven input-facilitated college oral English teaching model in oral English classroom. It is performed to realize the transformation of effective oral English teaching methods under the network environment. The experimental results show that the transformation effect of the effective teaching method of oral English, under the designed network environment, is better as compared to the existing work. The error detection rate in the proposed methodology reaches up to 98.6%, which is significantly higher than the results obtained by researchers in their existing work. The proposed method has obvious application advantages.

show abstract

Mouth2Audio: intelligible audio synthesis from videos with distinctive vowel articulation

Garg

Ruan

Hamarneh

et al. 2023

Int J Speech Technol

Self Cite

View full text Add to dashboard Cite

Humans use both auditory and facial cues to perceive speech, especially when auditory input is degraded, indicating a direct association between visual articulatory and acoustic speech information. This study investigates how well an audio signal of a word can be synthesized based on visual speech cues. Specifically, we synthesized audio waveforms of the vowels in monosyllabic English words from motion trajectories extracted from image sequences in the video recordings of the same words. The articulatory movements were recorded in two different speech styles: plain and clear. We designed a deep network trained on mouth landmark motion trajectories on a spectrogram and formant-based custom loss for different speech styles separately. Human and automatic evaluation show that our framework using visual cues can generate identifiable audio of the target vowels from distinct mouth landmark movements. Our results further demonstrate that intelligible audio can be synthesized from novel unseen talkers that were independent of the training data.

show abstract

Cross-linguistic perception of clearly spoken English tense and lax vowels based on auditory, visual, and auditory-visual information

Cited by 11 publications

References 51 publications

Plain-to-clear speech video conversion for enhanced intelligibility

Plain-to-clear speech video conversion for enhanced intelligibility

A Transformation of Oral English Teaching Method in the Network Environment

Mouth2Audio: intelligible audio synthesis from videos with distinctive vowel articulation

Contact Info

Product

Resources

About