Improved Mask-CTC for Non-Autoregressive End-to-End ASR

Abstract

For real-world deployment of automatic speech recognition (ASR), the system is desired to be capable of fast inference while relieving the requirement of computational resources. The recently proposed end-to-end ASR system based on mask-predict with connectionist temporal classification (CTC), Mask-CTC, fulfills this demand by generating tokens in a non-autoregressive fashion. While Mask-CTC achieves remarkably fast inference speed, its recognition performance falls behind that of conventional autoregressive (AR) systems. To boost the performance of Mask-CTC, we first propose to enhance the encoder network architecture by employing a recently proposed architecture called Conformer. Next, we propose new training and decoding methods by introducing auxiliary objective to predict the length of a partial target sequence, which allows the model to delete or insert tokens during inference. Experimental results on different ASR tasks show that the proposed approaches improve Mask-CTC significantly, outperforming a standard CTC model (15.5% → 9.1% WER on WSJ). Moreover, Mask-CTC now achieves competitive results to AR models with no degradation of inference speed (< 0.1 RTF using CPU). We also show a potential application of Mask-CTC to end-to-end speech translation.

Non-Autoregressive ASR with Self-Conditioned Folded Encoders

“…A key factor in improving the performance of CTCbased models is how to handle dependencies between tokens in a non-autoregressive manner. Approaches based on iterative refinement of the output tokens by token decoders are well known, and several methods have been proposed [11,12,13,14]. Mask-CTC [12] is a model that inputs the output of the CTC to a decoder and refines the tokens of low confidence conditioned on the tokens of high confidence, and Align-Refine [13] is a model that inputs the potential alignment into the decoder and refines the model on the alignment space.…”

Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chain

“…Inspired by the success of NAR models in NMT, several NAR methods were also proposed to reach the performance of AR models on ASR [16,17,18,19,20,21,22]. Since CTC learns a frame-wise latent alignment between the input speech and output tokens and predicts the target sequence based on a strong conditional independence assumption [23], it can be viewed as an early-stage realization of NAR ASR models.…”

“…In [18], Imputer was proposed to iteratively generate a new CTC alignment based on mask prediction. Besides, Mask-CTC [17,20] and Align-Refine [21] aimed to refine a tokenlevel CTC output or latent alignments with the mask prediction. In [19], Tian et al proposed to use the estimated CTC spikes to predict the length of target sequence and adopt the encoder states as the input of decoder.…”

An Improved Single Step Non-Autoregressive Transformer for Automatic Speech Recognition

“…To accelerate the inference, non-autoregressive transformers (NAT) were proposed for the parallel generation of the output sequence. The idea is widely adopted in neural machine translation (NMT) [4][5][6], automatic speech recognition (ASR) [7][8][9][10][11][12][13][14][15][16][17][18], text-to-speech (TTS) [19,20] and speech translation [21].…”

“…Essentially, autoregressive models are also iterative-based since they use a left-to-right generation order and take N iterations to generate a sequence of length N. Hence, the idea of iterative NAT is to adopt a different generation order with less than N iterations to accelerate the inference. Chen et al regarded the transformer decoder as a masked language model that first generates tokens with high confidence [7], while Higuchi et al applied the same idea but based on the connectionist temporal classification (CTC) output [11,16]. In addition, Fujita et al used the idea of the insertion transformer from NMT to generate the output sequence with an arbitrary order [12].…”