Accented Speech Recognition Based on End-to-End Domain Adversarial Training of Neural Networks

Abstract

The performance of automatic speech recognition (ASR) may be degraded when accented speech is recognized because the speech has some linguistic differences from standard speech. Conventional accented speech recognition studies have utilized the accent embedding method, in which the accent embedding features are directly fed into the ASR network. Although the method improves the performance of accented speech recognition, it has some restrictions, such as increasing the computational costs. This study proposes an efficient method of training the ASR model for accented speech in a domain adversarial way based on the Domain Adversarial Neural Network (DANN). The DANN plays a role as a domain adaptation in which the training data and test data have different distributions. Thus, our approach is expected to construct a reliable ASR model for accented speech by reducing the distribution differences between accented speech and standard speech. DANN has three sub-networks: the feature extractor, the domain classifier, and the label predictor. To adjust the DANN for accented speech recognition, we constructed these three sub-networks independently, considering the characteristics of accented speech. In particular, we used an end-to-end framework based on Connectionist Temporal Classification (CTC) to develop the label predictor, a very important module that directly affects ASR results. To verify the efficiency of the proposed approach, we conducted several experiments of accented speech recognition for four English accents including Australian, Canadian, British (England), and Indian accents. The experimental results showed that the proposed DANN-based model outperformed the baseline model for all accents, indicating that the end-to-end domain adversarial training effectively reduced the distribution differences between accented speech and standard speech.

Mitigating bias against non-native accents

“…Domain Adversarial Training (DAT) is a popular and common training strategy for domain adaptation [18,42]. The main assumption of DAT is that the training and test data come from different distributions such that an effective domain transfer from the training/source domain to the test/target domain is needed.…”

“…[16] proves that performing gradient reversal in domain adversarial training (DAT) is equivalent to minimizing the difference of output distributions of different accents. DAT [29] improved the performance of accented speech recognition for both end-to-end (E2E) [18] and hybrid ASR systems [29]. The experimental results in [29] showed that the performance of DAT was better than that of multi-task learning.…”

“…There is very limited research focusing on bias mitigation against non-native accents, and most existing research focuses on accented speech recognition. These studies [13][14][15][16][17][18] aim to improve the performance of non-native accented speech recognition, i.e., lowering the word error rate (WER). However, in order to build inclusive ASR for treating non-native speakers the same way as native speakers, i.e., ASR for everyone irrespective of how one 1 Introduction speaks or the accent one has, the aim should not be just to lower the WER, but also to pay close attention to and reduce the performance gap between non-accented and accented speech.…”

“…In addition to the straightforward solution: increasing the amount of the non-native accented speech data, we can also get inspired from previous accented speech recognition works. In order to make ASR systems more robust to non-native accents, various training methods have been proposed including contrastive learning [13], domain adversarial training (DAT) [14] [16] [18], multi-task learning [17] and transfer learning [14]. Specifically, both contrastive learning and DAT are domain adaption methods, which can be used to minimize the difference between the standard speech data and the non-native accented speech data.…”

Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptation

“…With this approach, it is believed that the output representations of the feature extractor can be domaininvariant, so the downstream model can perform comparable results in both source and target domains. [13][14][15][16] trained automatic speech recognition models to deal with accented speech with DAT. [17] proposed to train a multi-lingual speech emotion recognition model with adversarial domain adaptation.…”