Transformer Quality in Linear Time

Hua, Weizhe; Dai, Zihang; Liu, Hanxiao; Le, Quoc V.

doi:10.48550/arxiv.2202.10447

Cited by 8 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They combine the strengths of CNNs (efficient training), RNNs (efficient inference), and continuous models (robust to change in sampling rates). LambdaNetworks [2], AFT [90] and FLASH [40] are other attempts at replacing attention in the context of image classification and language modeling.…”

Section: A Related Workmentioning

confidence: 99%

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Dao¹,

Fu²,

Ermon³

et al. 2022

Preprint

View full text Add to dashboard Cite

Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length. Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup. We argue that a missing principle is making attention algorithms IOaware-accounting for reads and writes between levels of GPU memory. We propose FlashAttention, an IO-aware exact attention algorithm that uses tiling to reduce the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip SRAM. We analyze the IO complexity of FlashAttention, showing that it requires fewer HBM accesses than standard attention, and is optimal for a range of SRAM sizes. We also extend FlashAttention to block-sparse attention, yielding an approximate attention algorithm that is faster than any existing approximate attention method. FlashAttention trains Transformers faster than existing baselines: 15% end-to-end wall-clock speedup on BERT-large (seq. length 512) compared to the MLPerf 1.1 training speed record, 3× speedup on GPT-2 (seq. length 1K), and 2.4× speedup on long-range arena (seq. length 1K-4K). FlashAttention and block-sparse FlashAttention enable longer context in Transformers, yielding higher quality models (0.7 better perplexity on GPT-2 and 6.4 points of lift on long-document classification) and entirely new capabilities: the first Transformers to achieve better-than-chance performance on the Path-X challenge (seq. length 16K, 61.4% accuracy) and Path-256 (seq. length 64K, 63.1% accuracy).

show abstract

Section: A Related Workmentioning

confidence: 99%

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Dao¹,

Fu²,

Ermon³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In this investigation, we consider that both global and local interactions are important for being parameter efficient and examine how to improve computing efficiency. We propose that a novel combination of ResNet [20] and GAU [21] will achieve the best of both worlds-a deep residual convolutional framework can capture the relative-offset-based local correlations of audio sequences progressively via a local receptive field layer-by-layer whilst a simpler, yet more performant, layer than the other transformer architectures, is used to learn content-based global interactions. The remainder of this paper is as follows: Section 2 points out the difficulties in using speech recognition techniques in this field, and also expounds the efforts of other researchers in the application of the techniques for ATC.…”

Section: Introductionmentioning

confidence: 99%

Speech GAU: A Single Head Attention for Mandarin Speech Recognition for Air Traffic Control

et al. 2022

View full text Add to dashboard Cite

The rise of end-to-end (E2E) speech recognition technology in recent years has overturned the design pattern of cascading multiple subtasks in classical speech recognition and achieved direct mapping of speech input signals to text labels. In this study, a new E2E framework, ResNet–GAU–CTC, is proposed to implement Mandarin speech recognition for air traffic control (ATC). A deep residual network (ResNet) utilizes the translation invariance and local correlation of a convolutional neural network (CNN) to extract the time-frequency domain information of speech signals. A gated attention unit (GAU) utilizes a gated single-head attention mechanism to better capture the long-range dependencies of sequences, thus attaining a larger receptive field and contextual information, as well as a faster training convergence rate. The connectionist temporal classification (CTC) criterion eliminates the need for forced frame-level alignments. To address the problems of scarce data resources and unique pronunciation norms and contexts in the ATC field, transfer learning and data augmentation techniques were applied to enhance the robustness of the network and improve the generalization ability of the model. The character error rate (CER) of our model was 11.1% on the expanded Aishell corpus, and it decreased to 8.0% on the ATC corpus.

show abstract

“…Recently, Google proposed a new model architecture to address the quality and empirical speed issues of existing Transformer variants. This is achieved by combining the Attention layer and FFN into a single unit called GAU while reducing it to just one head (Hua et al, 2022). However, it is flawed in many details such as the scaling factor and the replacement of softmax.…”

Section: Introductionmentioning

confidence: 99%

“…

In February this year Google proposed a new Transformer variant called FLASH (Hua et al, 2022), which has a faster speed, lower VRAM footprint and better performance. This is achieved by designing a performant layer named GAU (Gated Attention Unit), which combines the Attention layer and FFN.

…”

mentioning

confidence: 99%

See 1 more Smart Citation

Supplementary Material: Implementation and Experiments for GAU-based Model

Liu¹

2022

Preprint

View full text Add to dashboard Cite

In February this year Google proposed a new Transformer variant called FLASH (Hua et al., 2022), which has a faster speed, lower VRAM footprint and better performance. This is achieved by designing a performant layer named GAU (Gated Attention Unit), which combines the Attention layer and FFN. In this paper, some implementation details are re-analyzed both theoretically and practically. We then propose a novel GAU-based model and pre-train it on a Chinese corpus. Results of the CLUE benchmark show that our model achieves a dev average score of 75.02, 1% higher than RoFormerV1 and being 45% faster, which is also competitive with Ro-FormerV2.

show abstract

Transformer Quality in Linear Time

Cited by 8 publications

References 0 publications

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Speech GAU: A Single Head Attention for Mandarin Speech Recognition for Air Traffic Control

Supplementary Material: Implementation and Experiments for GAU-based Model

Contact Info

Product

Resources

About