Efficient Transformer for Direct Speech Translation

Abstract

The advent of Transformer-based models has surpassed the barriers of text. When working with speech, we must face a problem: the sequence length of an audio input is not suitable for the Transformer. To bypass this problem, a usual approach is adding strided convolutional layers, to reduce the sequence length before using the Transformer.In this paper, we propose a new approach for direct Speech Translation, where thanks to an efficient Transformer we can work with a spectrogram without having to use convolutional layers before the Transformer. This allows the encoder to learn directly from the spectrogram and no information is lost. We have created an encoder-decoder model, where the encoder is an efficient Transformer -the Longformer-and the decoder is a traditional Transformer decoder. Our results, which are close to the ones obtained with the standard approach, show that this is a promising research direction.

Multiformer: A Head-Configurable Transformer-Based Model for Direct Speech Translation

“…First, it can be observed from Table 1, that the efficient architecture based only on Local Attention (local_attention) already obtains the same results as the baseline, suggesting the presence of unnecessary computations in self-attention. Unlike previous works (Alastruey et al, 2021), this architecture maintains the convolutional layers, so the amount of global content within the attention mechanism is higher using the same window size. On the other hand, while the architecture based exclusively on ConvAttention (conv_attention), man-3 More details in Table 2 in the Appendices.…”

“…However, after this compression, the resulting sequences are still considerably longer and more redundant than their text counterparts. Alastruey et al (2021) proposed the use of efficient Transformers for ST, but, as observed in different tasks by Tay et al (2021), they suffer from a drop in performance quality. The main reason for this deterioration is that most efficient Transformers propose strategies that deprive the model of the ability to learn all types of content from the input stream.…”

“…In particular, given a fixed window size w, each token attends to w 2 tokens on each side, achieving a linear scaling (O(n × w)) of the module complexity. As in Alastruey et al (2021), this attention pattern is intended to force the learning of local relations, while being more efficient.…”

“…For the ConvAttention, we use a kernel size of 5 and a stride of 2, reducing the length of keys and values to the half. Regarding the Local Attention, as in Alastruey et al (2021) we have chosen a window size of 64 tokens. These hyperparameters have been employed in all Multiformer architectures.…”

“…The closest work to ours was done by Alastruey et al (2021), who used Longformer's (Beltagy et al, 2020) local attention pattern as an efficient alternative to self-attention for speech processing. However, they observed that, due to the scarcity of global context in the encoder output, the quality of the translations was slightly hindered.…”