Efficient Weight factorization for Multilingual Speech Recognition

Abstract

End-to-end multilingual speech recognition involves using a single model training on a compositional speech corpus including many languages, resulting in a single neural network to handle transcribing different languages. Due to the fact that each language in the training data has different characteristics, the shared network may struggle to optimize for all various languages simultaneously. In this paper we propose a novel multilingual architecture that targets the core operation in neural networks: linear transformation functions. The key idea of the method is to assign fast weight matrices for each language by decomposing each weight matrix into a shared component and a language dependent component. The latter is then factorized into vectors using rank-1 assumptions to reduce the number of parameters per language. This efficient factorization scheme is proved to be effective in two multilingual settings with 7 and 27 languages, reducing the word error rates by 26% and 27% rel. for two popular architectures LSTM and Transformer, respectively.

Adaptive multilingual speech recognition with pretrained models

“…We also provided further analysis to the architecture, by showing that there are benefits to either improving the self-attention mechanism by adding relative positions during fine-tuning, or stacking the MBART encoders to the wav2vec counterpart. This is the continuation of the line of work building multilingual ASR systems based on sequence-to-sequence neural networks [17,5,16]. Our work is available for public at https://github.com/quanpn90/NMTGMinor providing a highly CUDA-optimized implementation for both wav2vec and MBART which is potentially useful for the community.…”

“…The motivation comes from the disbelief that there are features being shared between languages and at the same time each language requires to selectively represented, and networks are encouraged to change "modes" depending on the language being processed [23]. Since then, multingual model designers opt to use specific network components being presented for each language, ranging from weight generator [24] to adapters [15,22] or recently adaptive weights adding scales and biases to each weight matrix in the whole architecture [16]. In this paper, the last two options are selected for investigation thanks to being computationally manageable.…”

“…On the other hand, adaptive weights [16] was proposed based on the observation that neural networks evolve rapidly yet the core remains to be matrix multiplication. Therefore, it is possible to separate the weight matrix into a shared component WS and language dependent adaptive scale WML and bias WBL.…”

“…More importantly, there are many improvements for the Transformers over the years that improve either modeling long range dependencies in self-attention [14,10] or adaptive components for multilingual models [15,16]. It is promising to also combine these techniques with the pretrained models for the best results.…”

“…• Second, the language specific modulation techniques such as language adapters [15] and factorized adaptive weights [16] complement the two pretrained modules very well and have a strong impact on especially the lowresource languages mentioned above.…”