Self-Attention with Relative Position Representations

Shaw, Peter J.; Uszkoreit, Jakob; Vaswani, Ashish

doi:10.48550/arxiv.1803.02155

Cited by 284 publications

(415 citation statements)

References 0 publications

Supporting

Mentioning

414

Contrasting

Order By: Relevance

“…In apart from the network architecture modification, several advanced tricks for vision transformer are also adopted. Relative position encoding [25] is added on self-attention module to better represent relative position between tokens. Linear spatial reduction attention (LSRA) [33] is utilized in the first two stages to reduce the computation cost of self-attention for long sequence.…”

Section: Methodsmentioning

confidence: 99%

“…Inspired by the recent works [34,36], we introduce two main architecture modifications: 1) pyramid architecture with gradual decreased resolution to extract multi-scale representa- tions, and 2) convolutional stem for improving the patchify stem and stable training. We also include several other tricks [25,33] to further improve the efficiency. The new transformer is named as PyramidTNT.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture

Han¹,

Guo²,

Wang³

2022

Preprint

View full text Add to dashboard Cite

Transformer networks have achieved great progress for computer vision tasks. Transformer-in-Transformer (TNT) architecture utilizes inner transformer and outer transformer to extract both local and global representations. In this work, we present new TNT baselines by introducing two advanced designs: 1) pyramid architecture, and 2) convolutional stem. The new "PyramidTNT" significantly improves the original TNT by establishing hierarchical representations. PyramidTNT achieves better performances than the previous state-of-the-art vision transformers such as Swin Transformer. We hope this new baseline will be helpful to the further research and application of vision transformer. Code will be available at https: //github.com/huawei-noah/CV-Backbones/ tree/master/tnt_pytorch.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture

Han¹,

Guo²,

Wang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Because the added position embedding depends on the absolute positions of tokens in a sequence, it is called absolute position encoding. We'll use relative position encoding [12], which can directly encode the distance between tokens. To increase the accuracy of the short-term load forecast, we modified the similarity functions (Equation (1) and Equation ( 5)) as in…”

Section: Relative Position Encoding For Transformermentioning

confidence: 99%

Efficient Self-attention with Relative Position Encoding for Electric Power Load Forecasting

Chen

Yang

Liu

et al. 2022

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

To effectively mine historical data information and improve the accuracy of short-term load prediction, this paper aims at the characteristics of time series and nonlinear power load. Deep learning for load forecasting has received a lot of attention in recent years, and it has become popular in the analysis of electricity load forecasting. Long short-term memory (LSTM) and gated recurrent unit (GRU) are specifically designed for time-series data. However, due to the gradient disappearing and exploding problem, recurrent neural networks (RNNs) cannot capture long-term dependence. The Transformer, a self-attention-based sequence model, has produced impressive results in a variety of generating tasks that demand long-range coherence. This shows that self-attention could be useful in power load forecasting modeling. In this paper, to effectively and efficiently model the large-scale load forecasting, we further design the transform encoder with relative position encoding, which consists of four main components: single-layer neural network, relative positional encoding module, encoder module, and feed-forward network. Experimental results on real-world datasets demonstrate that our method outperforms the GRU, LSTM, and original Transformer encoder.

show abstract

“…This phenomenon could result from the inductive bias of CNNs, namely spatial similarity. In the future, we could add other popular modules to our model, such as absolute positional encoding [7], relative positional encoding [62], and conditional positional encoding [63], which could further improve the performance. We could also pre-train our model for few-shot or zero-shot learning, where only a few supervised labels are required in training.…”

Section: Time Seriesmentioning

confidence: 99%

Classification of Long Sequential Data using Circular Dilated Convolutional Neural Networks

Liu¹,

Khalitov²,

Tao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Classification of long sequential data is an important Machine Learning task and appears in many application scenarios. Recurrent Neural Networks, Transformers, and Convolutional Neural Networks are three major techniques for learning from sequential data. Among these methods, Temporal Convolutional Networks (TCNs) which are scalable to very long sequences have achieved remarkable progress in time series regression. However, the performance of TCNs for sequence classification is not satisfactory because they use a skewed connection protocol and output classes at the last position. Such asymmetry restricts their performance for classification which depends on the whole sequence. In this work, we propose a symmetric multi-scale architecture called Circular Dilated Convolutional Neural Network (CDIL-CNN), where every position has an equal chance to receive information from other positions at the previous layers. Our model gives classification logits in all positions, and we can apply a simple ensemble learning to achieve a better decision. We have tested CDIL-CNN on various long sequential datasets. The experimental results show that our method has superior performance over many state-of-the-art approaches. The model and experiments are available at https://github.com/LeiCheng-no/CDIL-CNN.

show abstract

Self-Attention with Relative Position Representations

Cited by 284 publications

References 0 publications

PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture

PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture

Efficient Self-attention with Relative Position Encoding for Electric Power Load Forecasting

Classification of Long Sequential Data using Circular Dilated Convolutional Neural Networks

Contact Info

Product

Resources

About