Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Text Data

Abstract

This paper presents a method to pre-train transformer-based encoder-decoder automatic speech recognition (ASR) models using sufficient target-domain text. During pre-training, we train the transformer decoder as a conditional language model with empty or artifical states, rather than the real encoder states. By this pre-training strategy, the decoder can learn how to generate grammatical text sequence before learning how to generate correct transcriptions. Contrast to other methods which utilize text only data to improve the ASR performance, our method does not change the network architecture of the ASR model or introduce extra component like textto-speech (TTS) or text-to-encoder (TTE). Experimental results on LibriSpeech corpus show that the proposed method can relatively reduce the word error rate over 10%, using 960 hours transcriptions.

Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data

“…In addition, an idea was explored to pre-train a decoder for end-to-end ASR [4,14,15]. The authors in [4] employ a single speaker text to speech (TTS) system to generate synthesized speech from a large number of transcripts, and use the generated speech-text pairs to pre-train the decoder.…”

“…The authors in [4] employ a single speaker text to speech (TTS) system to generate synthesized speech from a large number of transcripts, and use the generated speech-text pairs to pre-train the decoder. In [14], unpaired text data are used to pre-train the transformer decoder, which is pre-trained as a conditional language model by constructing empty or artificial states to replace the real encoder hidden states. Leveraging large-scale unpaired speech and text data, SpeechT5 [15] pre-trains a shared encoder-decoder model for various spoken language tasks.…”

“…However, many previous studies only pre-train the speech encoder for various spoken downstream tasks, while the decoder is not pre-trained for the end-to-end encoder-decoder based ASR models, which usually rely heavily on a large amount of transcribed audio data. Although there are some attempts to pre-train a Transformer decoder for end-to-end ASR model, they have to use additional unpaired text data [4,14,15]. Can we pre-train the ASR decoder with speech-only data?…”

Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model

“…Recently, how to use unpaired text data that is much easier to collect compared with paired speech-text data to improve S2S architecture has attracted more and more attention [33]. They try to solve the problem that the decoder depends on the encoder output and thus cannot be separately pretrained [13,34]. For example, Gao et al [13] pretrain the decoder on unpaired text data with empty or artificial states instead of real encoder states, which not only fails to use those already pretrained LMs like GPT2, but the result is very limited due to the input state mismatch between pre-training and fine-tuning.…”

“…Furthermore, the decoder of S2S ASR models cannot be pre-trained separately because of its dependency on acoustics representations. And it is hard to utilize pre-trained LMs like BERT [8] or GPT2 [12] for parameter initialization due to architecture mismatch [13]. Therefore, how to efficiently utilize the pretrained acoustic models (AMs) and language models (LMs) in dominant S2S ASR models still remains a valuable challenge.…”