On The Usefulness of Self-Attention for Automatic Speech Recognition with Transformers

Abstract

Self-attention models such as Transformers, which can capture temporal relationships without being limited by the distance between events, have given competitive speech recognition results. However, we note the range of the learned context increases from the lower to upper self-attention layers, whilst acoustic events often happen within short time spans in a left-to-right order. This leads to a question: for speech recognition, is a global view of the entire sequence useful for the upper self-attention encoder layers in Transformers? To investigate this, we train models with lower selfattention/upper feed-forward layers encoders on Wall Street Journal and Switchboard. Compared to baseline Transformers, no performance drop but minor gains are observed. We further developed a novel metric of the diagonality of attention matrices and found the learned diagonality indeed increases from the lower to upper encoder self-attention layers. We conclude the global view is unnecessary in training upper encoder layers.

Stochastic Attention Head Removal: A Simple and Effective Method for Improving Transformer Based ASR Models

“…We introduce the stochastic attention head removal (SAHR) strategy in this section. In our previous work [11], we found in trained Transformer ASR models, there are heads which only generates nearly diagonal attention matrices. The high diagonality indicates these heads are merely identity mappings and thus they are redundant.…”

“…Zhou et al [21] proposed a similar method of training selfattention models for natural language processing (NLP). However, the related work does not analyse where the benefits of such methods come from, whilst in contrast, we independently derive our method based on our previous work which studies the usefulness of the attention heads [11]. Additionally, in this work, our detailed analysis of the proposed method reveals why dynamically removing attention heads is helpful.…”

“…Since self-attention encodes sequences through attention mechanisms, it can model dependencies within any range. However, previous works have shown it is less effective in capturing local information [11,22]. Nevertheless, acoustic events usually happen within short periods and thus local information is essential for ASR tasks.…”

“…We use WSJ as the dataset and the experimental setups are presented in Section 5. For the baseline model, following our previous work, we plot the heatmap of the averaged diagonality [11] on WSJ eval92 for each head in every encoder layer. Then, we remove the heads whose averaged diagonality are above a threshold and retrain the model.…”

“…In self-attention based models such as Transformers [1], each self-attention layer uses multi-head attention to capture a set of different inputs representations, and have given competitive ASR results [2][3][4][5][6][7][8][9][10]. However, we previously observed that not all of the attention heads are useful [11]. We found that in trained Transformer ASR models, the attention matrices of some attention heads are close to the identity matrices, indicating that the attention mechanism is just an identity mapping.…”