Layer-Wise Fast Adaptation for End-to-End Multi-Accent Speech Recognition

Abstract

Accent variability has posed a huge challenge to automatic speech recognition (ASR) modeling. Although one-hot accent vector based adaptation systems are commonly used, they require prior knowledge about the target accent and cannot handle unseen accents. Furthermore, simply concatenating accent embeddings does not make good use of accent knowledge, which has limited improvements. In this work, we aim to tackle these problems with a novel layer-wise adaptation structure injected into the E2E ASR model encoder. The adapter layer encodes an arbitrary accent in the accent space and assists the ASR model in recognizing accented speech. Given an utterance, the adaptation structure extracts the corresponding accent information and transforms the input acoustic feature into an accent-related feature through the linear combination of all accent bases. We further explore the injection position of the adaptation layer, the number of accent bases, and different types of accent bases to achieve better accent adaptation. Experimental results show that the proposed adaptation structure brings 12% and 10% relative word error rate (WER) reduction on the AESRC2020 accent dataset and the Librispeech dataset, respectively, compared to the baseline.

Wav2vec‐MoE: An unsupervised pre‐training and adaptation method for multi‐accent ASR

“…proposed a MixNet‐based architecture to compensate for phonetic and accent variabilities by using MoE. Gong [8] et al. proposed a layer‐wise adapter in which each expert learns the features of the inputs and finally merges the features from others.…”

“…Introduction: As one of the essential technologies of human-computer interaction, end-to-end automatic speech recognition (ASR) has achieved remarkable performance in recent research [1][2][3]. While in real life, either the subjective factors of speakers or the objective environment degrade the performance of ASR [4][5][6][7][8]. This study focuses on one of the subjective factors-accented speech.…”

“…Jain [15] et al proposed a MixNetbased architecture to compensate for phonetic and accent variabilities by using MoE. Gong [8] et al proposed a layer-wise adapter in which each expert learns the features of the inputs and finally merges the features from others. These methods demonstrate the potential of MoE in multi-accent ASR, but their improvement is limited by limited annotated speech data.…”