Transformer-Based End-to-End Speech Recognition with Residual Gaussian-Based Self-Attention

Abstract

Self-attention (SA), which encodes vector sequences according to their pairwise similarity, is widely used in speech recognition due to its strong context modeling ability. However, when applied to long sequence data, its accuracy is reduced. This is caused by the fact that its weighted average operator may lead to the dispersion of the attention distribution, which results in the relationship between adjacent signals ignored. To address this issue, in this paper, we introduce relativeposition-awareness self-attention (RPSA). It not only maintains the global-range dependency modeling ability of self-attention, but also improves the localness modeling ability. Because the local window length of the original RPSA is fixed and sensitive to different test data, here we propose Gaussian-based selfattention (GSA) whose window length is learnable and adaptive to the test data automatically. We further generalize GSA to a new residual Gaussian self-attention (resGSA) for the performance improvement. We apply RPSA, GSA, and resGSA to Transformer-based speech recognition respectively. Experimental results on the AISHELL-1 Mandarin speech recognition corpus demonstrate the effectiveness of the proposed methods. For example, the resGSA-Transformer achieves a character error rate (CER) of 5.86% on the test set, which is relative 7.8% lower than that of the SA-Transformer. Although the performance of the proposed resGSA-Transformer is only slightly better than that of the RPSA-Transformer, it does not have to tune the window length manually.

Iterative pseudo-labeling methods for improving speech recognition

“…The Conformer [5,6] contrast proposed by Anmol Gulati combines convolution combines convolution with self-attention, where self-attention learns global interactions and convolution effectively captures local correlations based on relative offsets, resulting in more efficient results than using convolution or self-attention alone.…”

An End-to-End Air Writing Recognition Method Based on Transformer

“…(b) Parallel computing capability: The self-attention mechanism in the transformer enables direct interaction between representations at each input position and all other positions, thereby facilitating highly parallelized computation [43]. Traditional recurrent neural network (RNN) models may encounter computational efficiency limitations when handling lengthy sequences in air-writing.…”