Phone Merging For Code-Switched Speech Recognition

Abstract

Speakers in multilingual communities often switch between or mix multiple languages in the same conversation. Automatic Speech Recognition (ASR) of codeswitched speech faces many challenges including the influence of phones of different languages on each other. This paper shows evidence that phone sharing between languages improves the Acoustic Model performance for Hindi-English code-switched speech. We compare baseline system built with separate phones for Hindi and English with systems where the phones were manually merged based on linguistic knowledge. Encouraged by the improved ASR performance after manually merging the phones, we further investigate multiple data-driven methods to identify phones to be merged across the languages. We show detailed analysis of automatic phone merging in this language pair and the impact it has on individual phone accuracies and WER. Though the best performance gain of 1.2% WER was observed with manually merged phones, we show experimentally that the manual phone merge

Investigations on Speech Recognition Systems for Low-Resource Dialectal Arabic-English Code-Switching Speech

“…The global prevalence of this multilingual phenomenon has placed great demand over ASR systems to be able to handle such mixed input. Despite the increasing attention CS ASR has received in the past decade [21,95,11,102,91,87,103,63,64,30], it is still considered to be in a rather fledgling state, with only few language pairs covered, and a huge space for research investigations and advancements. One of the main bottlenecks hindering the development of CS NLP applications is the lack of resources [27].…”

“…The majority of the work done was based on the GMM-HMM model [21,95]. With the current advances in deep learning, researchers have directed their work towards DNN-based approaches, which have proven to outperform the traditional GMM-HMM models [91,87,102]. While hybrid systems have achieved great success, they require language-specific lexicons (which may not be present for low-resource languages) and linguistic knowledge.…”

“…The vast majority of the available CS speech corpora have covered Chinese-English [28,86,66,70,2], Hindi-English [34,82,87,90,78] and Spanish-English [88,33,82] language pairs. Less work has covered Arabic-English [54,46,47], Arabic-French [12,14], Frisian-Dutch [101], Mandarin-Taiwanese [69,68], Turkish-German [26], English-Malay [2], English-isiZulu [98] and Sepedi-English [73].…”

Exploring the use of Common Label Set to Improve Speech Recognition of Low Resource Indian Languages

“…Further, there is a strong character to sound correlation in the case of Indian languages. The vowels and consonants from seven Indian languages and their corresponding phoneme representations is shown in Table 1 1 . It can be seen that the first alphabet in all the seven languages correspond to phoneme "a", the second alphabet to "aa" so on.…”

“…Previously in conventional and end to end Automatic Speech recognition (ASR) with phone-based units (both context-dependent and context-independent), merging of phones across languages was discussed for code switched ASR in Hindi-English [1,2]. 1 The Phoneme mapping shown is based on: https://www.iitm.ac.in/donlab/tts/downloads/cls/cls v2. 1.6.pdf With similar motivation, a unified parser [3] was developed to generate lexicon for conventional HMM-DNN based Text-To-Speech Synthesis (TTS) in Indian languages.…”