Leveraging Text Data Using Hybrid Transformer-LSTM Based End-to-End ASR in Transfer Learning

Abstract

In this work, we study leveraging extra text data to improve lowresource end-to-end ASR under cross-lingual transfer learning setting. To this end, we extend our prior work [1], and propose a hybrid Transformer-LSTM based architecture. This architecture not only takes advantage of the highly effective encoding capacity of the Transformer network but also benefits from extra text data due to the LSTM-based independent language model network. We conduct experiments on our in-house Malay corpus which contains limited labeled data and a large amount of extra text. Results show that the proposed architecture outperforms the previous LSTM-based architecture [1] by 24.2% relative word error rate (WER) when both are trained using limited labeled data. Starting from this, we obtain further 25.4% relative WER reduction by transfer learning from another resource-rich language. Moreover, we obtain additional 13.6% relative WER reduction by boosting the LSTM decoder of the transferred model with the extra text data. Overall, our best model outperforms the vanilla Transformer ASR by 11.9% relative WER. Last but not least, the proposed hybrid architecture offers much faster inference compared to both LSTM and Transformer architectures.

SUH_ASR@LT-EDI-ACL2022: Transformer based Approach for Speech Recognition for Vulnerable Individuals in Tamil

“…Using the Hybrid based LSTM transformer, the WER is reduced with 25.4% by transfer learning. Additionally, 13% WER is reduced by LSTM decoder (Zeng et al, 2021). Transformer model encoding and decoding can be carried with self-attention and multi-head attention layer (Lee et al, 2021).…”

Earthquake Nowcasting with Deep Learning

“…Here, we use for the science transformer mode a simple attention mechanism in an initial decoder but use a recurrent net LSTM for the encoder as shown in Figure 13. Such mixtures have been investigated and compared [74,75]. We compare the two architectures shown in Figure 13a,b.…”

Exploring the conformational ensembles of protein-protein complex with transformer-based generative model

“…Deep learning and generative artificial intelligence have recently made unprecedented advances in computational biology 11,24,25 . Generative deep learning has been successful for the sampling of protein conformations 25 , such as long short-term memory (LSTM) 26 , autoencoders (AEs) 27,28 , variational autoencoders (VAEs) 29–33 , generative adversarial networks (GANs) 31,34 , score-based models 35 , energy-based models 36 , Transformer 31,37,38 , and active learning 33 . Training of deep generative models relies on conformations extracted from molecular dynamics simulations.…”