Transformer-Based Online CTC/Attention End-To-End Speech Recognition Architecture

Abstract

Recently, Transformer has gained success in automatic speech recognition (ASR) field. However, it is challenging to deploy a Transformer-based end-to-end (E2E) model for online speech recognition. In this paper, we propose the Transformer-based online CTC/attention E2E ASR architecture, which contains the chunk self-attention encoder (chunk-SAE) and the monotonic truncated attention (MTA) based self-attention decoder (SAD). Firstly, the chunk-SAE splits the speech into isolated chunks. To reduce the computational cost and improve the performance, we propose the state reuse chunk-SAE. Sencondly, the MTA based SAD truncates the speech features monotonically and performs attention on the truncated features. To support the online recognition, we integrate the state reuse chunk-SAE and the MTA based SAD into online CTC/attention architecture. We evaluate the proposed online models on the HKUST Mandarin ASR benchmark and achieve a 23.66% character error rate (CER) with a 320 ms latency. Our online model yields as little as 0.19% absolute CER degradation compared with the offline baseline, and achieves significant improvement over our prior work on Long Short-Term Memory (LSTM) based online E2E models.

Head-Synchronous Decoding for Transformer-Based Streaming ASR

“…A similar Transformer architecture is adopted for all the three tasks. The online encoder is similar to the one presented in [22], which is a 6-layer chunk-SAE. The sizes of central, left and right We conduct CTC/attention joint training with the CTC weight of 0.3 for all tasks.…”

“…( 1), the attention is computed upon the full sequence of encoder and/or decoder states as required by the softmax function, which poses a big challenge for online recognition. To stream the Transformer ASR system, chunk-hopping based strategies [8,20,21,22] have been applied on the encoder side, where the input utterance is spliced into overlapping chunks and the chunks are chronologically fed to the SAE. Thus, the latency of the online encoder is subject to the chunk size.…”

“…Also, a second-pass soft attention is performed on a small chunk of encoder states that end at the attended timestep, in order to add flexibility to the speech-to-text alignment. Another attempt to enable online decoding is based on Monotonic Truncated Attention (MTA) [22], which simplifies MoChA's training criterion and rules out the second-pass soft attention mechanism.…”

End-to-end Speech Recognition with Adaptive Computation Steps

“…In terms of output labels, WSJ and AiShell-1 have 52 and 4231 classes, respectively. The online Transformer model adopts a chunkwise self-attention encoder (chunk-SAE) as presented in [25]. Non-overlapping chunks with length Nc are spliced from the original utterance so that they could be sequentially fed into the model.…”

Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Text Data

“…Transformer-based E2E model is an encoder-decoder framework which has shown good performance for many ASR tasks [13,14,5]. Different from other RNN-based encoderdecoder model, transformer-based model uses multi-head attention (MHA) mechanism [15] to learn relationships between distant concepts, rather than relying on recurrent connections and memory cells.…”