Code-Switch Speech Rescoring with Monolingual Data

Abstract

In the automatic speech recognition (ASR) system, how to solve the problem of code-switch speech recognition has been a concern. Code-switch speech recognition is challenging due to data scarcity as well as diverse syntactic structures across languages. In this paper, we focus on the code-switch speech recognition in mainland China, which is obviously different from the Hong Kong and Southeast Asia area in linguistic characteristics. We propose a novel approach that only uses monolingual data for code-switch second-pass speech recognition which is also named language model rescoring. The approach converts the code-switch sentence to a monolingual sentence by a word mapping and language model determination step, therefore the issue of data scarcity is unnecessary to be considered. The word pairs during the word mapping step are generated by a fine-designed generation process that incorporates machine translation, word alignment, etc. We show that the proposed approach achieves an over 7.23% relative WER reduction from the naive monolingual language model (MLM) rescoring in our test set.

Personalization for BERT-based Discriminative Speech Recognition Rescoring

“…For each hypothesis, the second pass models calculate a score (typically, negative loglikelihood) that is combined with the first pass scores to determine the final score [4,5]. Scores can be computed using a language model [4,5,6], a masked language model [7,8], or other hypothesis-level models [9,10]. Rescoring performance has been improved by using contextual information in recent literature.…”

Towards Zero-Shot Code-Switched Speech Recognition

“…However, most of these large scale models skew towards highresourced languages [7] and do not seek to directly optimize for intra-sentential CS ASR between particular language pairs. A more promising direction towards zero-shot CS ASR can be found in prior works which seek to incorporate monolingual data directly to improve CS performance [12][13][14][15][16][17][18][19][20][21][22]. In particular, there are several works which achieve joint modeling of CS and monolingual ASR by conditionally factorizing the overall bilingual task into monolingual parts [23][24][25].…”

Towards Zero-Shot Code-Switched Speech Recognition

“…However, most of these large scale models skew towards highresourced languages [9] and do not seek to directly optimize for intra-sentential CS ASR between particular language pairs. A more promising direction towards zero-shot CS ASR can be found in prior works which seek to incorporate monolingual data directly to improve CS performance [14][15][16][17][18][19][20][21][22][23][24][25][26][27][28]. In particular, there are several works which achieve joint modeling of CS and monolingual ASR by conditionally factorizing the overall bilingual task into monolingual parts [29][30][31].…”