Transformers in Vision: A Survey

Abstract

Astounding results from Transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems. Among their salient benefits, Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence as compared to recurrent networks e.g. , Long short-term memory (LSTM). Different from convolutional networks, Transformers require minimal inductive biases for their design and are naturally suited as set-functions. Furthermore, the straightforward design of Transformers allows processing multiple modalities ( e.g. , images, videos, text and speech) using similar processing blocks and demonstrates excellent scalability to very large capacity networks and huge datasets. These strengths have led to exciting progress on a number of vision tasks using Transformer networks. This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline. We start with an introduction to fundamental concepts behind the success of Transformers i.e., self-attention, large-scale pre-training, and bidirectional feature encoding. We then cover extensive applications of transformers in vision including popular recognition tasks ( e.g. , image classification, object detection, action recognition, and segmentation), generative modeling, multi-modal tasks ( e.g. , visual-question answering, visual reasoning, and visual grounding), video processing ( e.g. , activity recognition, video forecasting), low-level vision ( e.g. , image super-resolution, image enhancement, and colorization) and 3D analysis ( e.g. , point cloud classification and segmentation). We compare the respective advantages and limitations of popular techniques both in terms of architectural design and their experimental value. Finally, we provide an analysis on open research directions and possible future works. We hope this effort will ignite further interest in the community to solve current challenges towards the application of transformer models in computer vision.

Automated Recognition Of Pain In Cats

“…At this stage, we can conclude that using Resnet as deep learning backbone does not outperform humans in feature selection. This, however, can possibly be improved by using more sophisticated network architectures and pipelines such transformers 48 .…”

Attention mechanisms in computer vision: A survey

“…Chaudhari et al [141] provided a survey of attention models in deep neural networks which concentrates on their application to natural language processing, while our work focuses on computer vision. Three more specific surveys [142][143][144] summarize the development of visual transformers while our paper reviews attention mechanisms in vision more generally, not just selfattention mechanisms. Wang and Tax [145] presented a survey of attention models in computer vision, but it only considers RNN-based attention models, which form just a part of our survey.…”

“…A detailed survey of vision transformers is omitted here as other recent surveys [142][143][144]171] comprehensively review the use of transformer methods for visual tasks.…”

IntFormer: Predicting pedestrian intention with the aid of the Transformer architecture

“…This architecture was proposed in [18] for NLP tasks, outperforming recurrent and convolutional models, which were state-of-the-art at that moment. Recently, several works have been published looking for ways to apply them to computer vision tasks [19]. This architecture relies on a self-attention mechanism, but, instead of using recurrence, transformer models create relationships between all samples in the input sequence, allowing parallelization and better use of modern devices such as TPUs and GPUs.…”