Independent Language Modeling Architecture for End-To-End ASR

Abstract

The attention-based end-to-end (E2E) automatic speech recognition (ASR) architecture allows for joint optimization of acoustic and language models within a single network. However, in a vanilla E2E ASR architecture, the decoder sub-network (subnet), which incorporates the role of the language model (LM), is conditioned on the encoder output. This means that the acoustic encoder and the language model are entangled that doesn't allow language model to be trained separately from external text data. To address this problem, in this work, we propose a new architecture that separates the decoder subnet from the encoder output. In this way, the decoupled subnet becomes an independently trainable LM subnet, which can easily be updated using the external text data. We study two strategies for updating the new architecture. Experimental results show that, 1) the independent LM architecture benefits from external text data, achieving 9.3% and 22.8% relative character and word error rate reduction on Mandarin HKUST and English NSC datasets respectively; 2) the proposed architecture works well with external LM and can be generalized to different amount of labelled data.

Leveraging Text Data Using Hybrid Transformer-LSTM Based End-to-End ASR in Transfer Learning

“…A LSTM-based encoder-decoder architecture [1], denoted as A1 in the rest of this paper, consists of a Bidirectional LSTM encoder and a LSTM-based decoder which are shown in Fig. 1.…”

“…Such techniques not only require external language models but also lead to a slow inference. To tackle this problem, [1] has proposed long short term memory (LSTM)-based encoderdecoder architecture which allows improving the LM capacity of the decoder using the extra text data. However, it utilized the LSTM structure for the encoder which has shown limited modeling capacity as well as slow training.…”

“…In this work, we propose a hybrid Transformer-LSTM architecture which combines the advantages of [1] and [6]. It not only has a high encoding capacity of the Transformer but also benefits from the extra text data due to the LSTM-based independent language model decoder.…”

“…The paper is organized as follows. Section 2 describes baseline architectures mentioned in [1] and [6]. Then, the proposed techniques are presented in Section 3.…”

“…

In this work, we study leveraging extra text data to improve lowresource end-to-end ASR under cross-lingual transfer learning setting. To this end, we extend our prior work [1], and propose a hybrid Transformer-LSTM based architecture. This architecture not only takes advantage of the highly effective encoding capacity of the Transformer network but also benefits from extra text data due to the LSTM-based independent language model network.

…”