Multilingual Speech Recognition with Self-Attention Structured Parameterization

Zhu, Yun; Haghani, Parisa; Tripathi, Anshuman; Ramabhadran, Bhuvana; Farris, Brian; Xu, Hainan; Han, Lu; Sak, Haşim; Leal, Isabel; Gaur, Neeraj; Moreno, Pedro J.; Zhang, Qian

doi:10.21437/interspeech.2020-2847

Cited by 20 publications

(17 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the one hand, the implementations often depends heavily on a certain architecture being popular at the time, and the given improvement is going to be diminished when a new architecture evolves. For example, the language-specifically biased attention [5] modified the self-attention architecture [6] specifically based on the assumption that each language can benefit from a bias added to the attention scores. On the other hand, the language-dependent components might require a considerable amount of parameters and struggles to scale to the num-ber of languages.…”

Section: Introductionmentioning

confidence: 99%

Efficient Weight factorization for Multilingual Speech Recognition

Pham¹,

Nguyen²,

Stueker³

et al. 2021

Preprint

View full text Add to dashboard Cite

End-to-end multilingual speech recognition involves using a single model training on a compositional speech corpus including many languages, resulting in a single neural network to handle transcribing different languages. Due to the fact that each language in the training data has different characteristics, the shared network may struggle to optimize for all various languages simultaneously. In this paper we propose a novel multilingual architecture that targets the core operation in neural networks: linear transformation functions. The key idea of the method is to assign fast weight matrices for each language by decomposing each weight matrix into a shared component and a language dependent component. The latter is then factorized into vectors using rank-1 assumptions to reduce the number of parameters per language. This efficient factorization scheme is proved to be effective in two multilingual settings with 7 and 27 languages, reducing the word error rates by 26% and 27% rel. for two popular architectures LSTM and Transformer, respectively.

show abstract

Section: Introductionmentioning

confidence: 99%

Efficient Weight factorization for Multilingual Speech Recognition

Pham¹,

Nguyen²,

Stueker³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Although it is simple and maximizes the sharing across languages, it also brings confusion between languages during recognition. If the multilingual model can condition the encoder on the language identity (LID), significant improvement can be obtained over the universal multilingual model without LID because the one-hot LID guides the ASR model to generate the transcription of the target language by reducing the confusion from other languages [127,[129][130][131][132][133]. However, such a multilingual E2E model with LID relies on the prior knowledge of which language the user will speak for every utterance, working more like a monolingual model.…”

Section: Multilingual Modelingmentioning

confidence: 99%

Recent Advances in End-to-End Automatic Speech Recognition

Li¹

2021

Preprint

View full text Add to dashboard Cite

Recently, the speech community is seeing a significant trend of moving from deep neural network based hybrid modeling to end-to-end (E2E) modeling for automatic speech recognition (ASR). While E2E models achieve the state-of-the-art results in most benchmarks in terms of ASR accuracy, hybrid models are still used in a large proportion of commercial ASR systems at the current time. There are lots of practical factors that affect the production model deployment decision. Traditional hybrid models, being optimized for production for decades, are usually good at these factors. Without providing excellent solutions to all these factors, it is hard for E2E models to be widely commercialized. In this paper, we will overview the recent advances in E2E models, focusing on technologies addressing those challenges from the industry's perspective.

show abstract

“…For all of these languages, the training and test data are anonymized and transcribed by humans. Similar to [9], we use two forms of data augmentation to mitigate overfitting and improve generalization, namely noise and reverberation based augmentation detailed in [17] and SpecAugment [18].…”

Section: Languages and Datamentioning

confidence: 99%

“…A variety of approaches have been explored for changing the structure of the neural network model to make it more amenable to multilingual modeling. In the context of encoderdecoder models, [5] used adapter layers to account for different amounts of available data per language, [9] parameterized the attention heads of a Transformer-based encoder to be per-language, while [6] showed that a multi-decoder multilingual model, where each decoder is assigned to a cluster of languages, can achieve good performance. Since the introduction of Mixture of Experts (MOE) in [10], these models have found popularity in machine translation [8], and speech recognition [11,12].…”

Section: Introductionmentioning

confidence: 99%

Mixture of Informed Experts for Multilingual Speech Recognition

Gaur

Farris

Haghani

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

When trained on related or low-resource languages, multilingual speech recognition models often outperform their monolingual counterparts. However, these models can suffer from loss in performance for high resource or unrelated languages. We investigate the use of a mixture-of-experts approach to assign per-language parameters in the model to increase network capacity in a structured fashion. We introduce a novel variant of this approach, 'informed experts', which attempts to tackle inter-task conflicts by eliminating gradients from other tasks in these task-specific parameters. We conduct experiments on a real-world task with English, French and four dialects of Arabic to show the effectiveness of our approach. Our model matches or outperforms the monolingual models for almost all languages, with gains of as much as 31% relative. Our model also outperforms the baseline multilingual model for all languages by up to 9% relative.Index Termsend-to-end speech recognition, multilingual, RNN-T, language id, mixture of experts * The first two authors have equal contribution. The rest of the list is sorted alphabetically.variations in amounts of training data. In this paper, we propose one multilingual model to transcribe languages with varied amounts of training data. We use the mixture-of-experts approach (MOE) [8] and adapt it to exploit the inherent structure of the data to simultaneously learn per-language experts.

show abstract

Multilingual Speech Recognition with Self-Attention Structured Parameterization

Cited by 20 publications

References 16 publications

Efficient Weight factorization for Multilingual Speech Recognition

Efficient Weight factorization for Multilingual Speech Recognition

Recent Advances in End-to-End Automatic Speech Recognition

Mixture of Informed Experts for Multilingual Speech Recognition

Contact Info

Product

Resources

About