Efficient Conformer with Prob-Sparse Attention Mechanism for End-to-End Speech Recognition

Abstract

End-to-end models are favored in automatic speech recognition (ASR) because of their simplified system structure and superior performance. Among these models, Transformer and Conformer have achieved state-of-the-art recognition accuracy in which self-attention plays a vital role in capturing important global information. However, the time and memory complexity of self-attention increases squarely with the length of the sentence. In this paper, a prob-sparse self-attention mechanism is introduced into Conformer to sparse the computing process of self-attention in order to accelerate inference speed and reduce space consumption. Specifically, we adopt a Kullback-Leibler divergence based sparsity measurement for each query to decide whether we compute the attention function on this query. By using the prob-sparse attention mechanism, we achieve impressively 8% to 45% inference speed-up and 15% to 45% memory usage reduction of the self-attention module of Conformer Transducer while maintaining the same level of error rate.

Deep Sparse Conformer for Speech Recognition

“…The range of top-u approximately holds as proved in [7]. Under the long tail distribution, it has been empirically testified that randomly sample U = LlogL dot-product pairs to compute the query sparsity measure and then select sparse top-u from it and form Q yielded acceptable results [9].…”

“…There are a list of work that proposed new attention mechanisms to replace the O(L 2 ) time/space complexities into O(LlogL) or even O(L) [8,9]. Motivated by [7] for modeling long sequences for time-series forecasting, we adapt the ProbSparse self-attention mechanism to replace the MHSA function in Equation 2.…”

“…We follow this sparse attention [7,8,9] direction and use a probability attention which defines a query importance measure in MHSA and then picks only the top-u (=O(logL)) query vectors for inner-product similarity computing with key vectors. For the other deeper Conformer direction, we borrow the deep normalization idea used for training 1,000-layer Transformers [10].…”

Similarity and Content-based Phonetic Self Attention for Speech Recognition

“…Architectural modifications for Transformer-based ASR models have been of great interest. Many works focus on reducing the heavy computational cost caused by SA [20,21,22,23]. For example, Efficient Conformer [20] proposed grouped SA and downsampling block to shorten the length of the sequence to be processed.…”

Conformer-based Hybrid ASR System for Switchboard Dataset

“…It uses * Equal contribution a convolution module to capture local context dependencies in addition to the long context captured by the self-attention module. The conformer architecture was investigated for different end-to-end systems such as attention encoder-decoder models [12,13], and recurrent neural network transducer [10,14]. Nevertheless, there has been no work investigating the impact of using a conformer AM for hybrid ASR systems.…”