Pronunciation augmentation for Mandarin-English code-switching speech recognition

Abstract

Code-switching (CS) refers to the phenomenon of using more than one language in an utterance, and it presents great challenge to automatic speech recognition (ASR) due to the code-switching property in one utterance, the pronunciation variation phenomenon of the embedding language words and the heavy training data sparse problem. This paper focuses on the Mandarin-English CS ASR task. We aim at dealing with the pronunciation variation and alleviating the sparse problem of code-switches by using pronunciation augmentation methods. An English-to-Mandarin mix-language phone mapping approach is first proposed to obtain a language-universal CS lexicon. Based on this lexicon, an acoustic data-driven lexicon learning framework is further proposed to learn new pronunciations to cover the accents, mis-pronunciations, or pronunciation variations of those embedding English words. Experiments are performed on real CS ASR tasks. Effectiveness of the proposed methods are examined on all of the conventional, hybrid, and the recent end-to-end speech recognition systems. Experimental results show that both the learned phone mapping and augmented pronunciations can significantly improve the performance of code-switching speech recognition.

The impact of student learning aids on deep learning and mobile platform on learning behavior

“…The language recognition system is used to realize the recognition of mixed-language speech in a single-language system. Long et al. (2021) pointed out that in order to make good use of a large amount of unlabeled data, semi-supervised learning is used to optimize the pronunciation dictionary, acoustic model and language model in the Chinese–English mixed speech recognition system.…”

“…The language recognition system is used to realize the recognition of mixed-language speech in a single-language system. Long et al (2021) pointed out that in order to make good use of a large amount of unlabeled data, semi-supervised learning is used to optimize the pronunciation dictionary, acoustic model and language model in the Chinese-English mixed speech recognition system. Part of the label mismatch data in the data set is treated as unsupervised data, and the standard Chinese and English pronunciation dictionary is modified in a semi-supervised manner, thereby solving the related problems caused by the spoken population.…”