Phoneme Set Design Based on Integrated Acoustic and Linguistic Features for Second Language Speech Recognition

Abstract

Recognition of second language (L2) speech is a challenging task even for state-of-the-art automatic speech recognition (ASR) systems, partly because pronunciation by L2 speakers is usually significantly influenced by the mother tongue of the speakers. Considering that the expressions of non-native speakers are usually simpler than those of native ones, and that second language speech usually includes mispronunciation and less fluent pronunciation, we propose a novel method that maximizes unified acoustic and linguistic objective function to derive a phoneme set for second language speech recognition. The authors verify the efficacy of the proposed method using second language speech collected with a translation game type dialogue-based computer assisted language learning (CALL) system. In this paper, the authors examine the performance based on acoustic likelihood, linguistic discrimination ability and integrated objective function for second language speech. Experiments demonstrate the validity of the phoneme set derived by the proposed method.

Classification of Utterances Based on Multiple BLEU Scores for Translation-Game-Type CALL Systems

“…We used a learner corpus as reference material, which consists of 2,803 utterances of 41 Japanese-speaking students and one bilingual English/Japanese speaker [21], [24]. Each Japanese student orally produced 66 utterances in English responding to 66 questions in situations such as shopping and making hotel reservations.…”