Intermediate Loss Regularization for CTC-Based Speech Recognition

Abstract

We present a simple and efficient auxiliary loss function for automatic speech recognition (ASR) based on the connectionist temporal classification (CTC) objective. The proposed objective, an intermediate CTC loss, is attached to an intermediate layer in the CTC encoder network. This intermediate CTC loss well regularizes CTC training and improves the performance requiring only small modification of the code and small and no overhead during training and inference, respectively. In addition, we propose to combine this intermediate CTC loss with stochastic depth training, and apply this combination to a recently proposed Conformer network. We evaluate the proposed method on various corpora, reaching word error rate (WER) 9.9% on the WSJ corpus and character error rate (CER) 5.2% on the AISHELL-1 corpus respectively, based on CTC greedy search without a language model. Especially, the AISHELL-1 task is comparable to other state-of-the-art ASR systems based on autoregressive decoder with beam search.

Layer Pruning on Demand with Intermediate CTC

“…Intermediate CTC [22] is an auxiliary loss designed for CTC modeling. It regularizes the model using an additional CTC loss attached at the intermediate layer of the encoder.…”

“…The only additional cost is to compute CTC loss for the given representation, which is much smaller than the cost of the encoder. [22] explores various choices for the intermediate layer positions, and concludes that it is sufficient to use one layer (K = 1) in the middle (l1 = L/2 ) for the regularization purpose. We revisit the effect of variants at Section 3.…”

“…The position variant of intermediate CTC is first explored by [22], concluding that its impact to the accuracy is minimal. However, we find the variant gives big difference on layer similarity and layer pruning.…”

“…Also, for high-end devices equipped with dedicated processors, CTC with modern architectures (e.g., Transformer [18,19] and Conformer [20,21]) allows fast parallel inference, coming from the non-autoregressive property of CTC. Also, while CTC has been regarded weaker than encoder-decoder, various ways to improve CTC, including pretraining [12] and regularization [22,23], have been developed. Also, there are active researches on CTC variants [12] and on non-autoregressive modeling based on CTC [24,25,26], suggesting that effective pruning for CTC can also be applied to other variants or non-autoregressive models.…”

“…In this work, we consider the on-demand layer pruning problem: training a deep neural network and removing some of the layers, thus reducing the depth of the network, without any fine-tuning. To achieve this, we employ two methods, intermediate CTC [22] and stochastic depth [27,28,5]. Although they are originally introduced as regularizers during training, we show they can be applied to layer pruning method as well.…”