Code-Switching Detection Using ASR-Generated Language Posteriors

Abstract

Code-switching (CS) detection refers to the automatic detection of language switches in code-mixed utterances. This task can be achieved by using a CS automatic speech recognition (ASR) system that can handle such language switches. In our previous work, we have investigated the code-switching detection performance of the Frisian-Dutch CS ASR system by using the time alignment of the most likely hypothesis and found that this technique suffers from over-switching due to numerous very short spurious language switches. In this paper, we propose a novel method for CS detection aiming to remedy this shortcoming by using the language posteriors which are the sum of the framelevel posteriors of phones belonging to the same language. The CS ASR-generated language posteriors contain more complete language-specific information on frame level compared to the time alignment of the ASR output. Hence, it is expected to yield more accurate and robust CS detection. The CS detection experiments demonstrate that the proposed language posterior-based approach provides higher detection accuracy than the baseline system in terms of equal error rate. Moreover, a detailed CS detection error analysis reveals that using language posteriors reduces the false alarms and results in more robust CS detection.

Multilingual self-supervised speech representations improve the speech recognition of low-resource African languages with codeswitching

“…In speech processing, work on code-switching can be divided into code-switching detection (Rallabandi et al, 2018;Yılmaz et al, 2016;Wang et al, 2019) using language identification (Choud-hury et al, 2017) andend-to-end recognition (Indra Winata et al, 2018). In this work, we look at both methods via finetuning of self-supervised representations, namely wav2vec 2.0 (Baevski et al, 2020).…”

The Royalflush System of Speech Recognition for M2met Challenge

“…Additionally, self-supervised fine-tuning methods are also proposed [9,10]. To solve the phonemic confusion issues, the Mixture of Experts (MoE) has become the mainstream model architecture for CS ASR system due to its ability not only to make use of contextual information between different languages, but also to discriminate different language information and extract language-specific representation [11][12][13][14].…”

“…Switch-Transformer-based MoE is another novel MoE model architecture, which not only mitigates the additional computational cost issue appeared in bi-encoder-based MoE, but also achieves better recognition performance [13,14,20,21]. However, the prior works still have some drawbacks, either ignoring language information or failing to achieve a real-time streaming ASR system.…”

“…However, incorporating language information could further enhance their accuracy. One approach, exemplified by [13], utilizes a Language Identification (LID) network with CTC loss to achieve good recognition performance. However, its reliance on a simplified, rule-based alignment strategy for handling blank tokens hinders real-time streaming functionality.…”

“…In CS and multilingual ASR, both the utilization of the contextual information across different languages and the extraction of the language-specific representations contribute to achieve good recognition performance [13,20]. Compared with the bi-encoder-based MoE with a shared encoder block, ST not only has the ability to boost the contextual information interaction via shared parts and extract language-specific representations through specialized parts but also requires lower computational costs [13]. So, LR-MoE [13] adopted ST as the foundation of CS and multilingual ASR, and significantly improved recognition performance over standard Transformer models with comparable computational complexity.…”