Visual-speech Synthesis of Exaggerated Corrective Feedback

Abstract

To provide more discriminative feedback for the second language (L2) learners to better identify their mispronunciation, we propose a method for exaggerated visual-speech feedback in computerassisted pronunciation training (CAPT). The speech exaggeration is realized by an emphatic speech generation neural network based on Tacotron, while the visual exaggeration is accomplished by ADC Viseme Blending, namely increasing Amplitude of movement, extending the phone's Duration and enhancing the color Contrast. User studies show that exaggerated feedback outperforms non-exaggerated version on helping learners with pronunciation identification and pronunciation improvement.

PTeacher: a Computer-Aided Personalized Pronunciation Training System with Exaggerated Audio-Visual Corrective Feedback

“…Numerous studies have suggested that many L2 speech production (pronunciation) difficulties are rooted in perception [15,22,23,60,76]. Moreover, it has been exemplified that reinforcing the perception ability of learners can significantly contribute to the speech production ability automatically [4,8,43,44,70,80]. Exaggerated audiovisual feedback is a particular kind of perception reinforcement, which corrects the pronunciation by strengthening the user's visual or auditory attention.…”