MViTv2: Improved Multiscale Vision Transformers for Classification and Detection

Li, Yanghao; Wu, Chao-Yuan; Fan, Haoqi; Mangalam, Karttikeya; Xiong, Bin; Malik, Jitendra; Feichtenhofer, Christoph

doi:10.48550/arxiv.2112.01526

Cited by 32 publications

(61 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They have also been generalized from the image to video domain [3,22,51,54]. In this work, we build our architecture based on the Multiscale Vision Transformer (MViT) architecture [22,44] as a con-crete instance, but the general idea can be applied to other ViT-based video models.…”

Section: Related Workmentioning

confidence: 99%

“…In this paper, we build MeMViT based on the MViT [22,44] architecture due to its strong performance, but the techniques presented in this paper can be applied to other ViTbased architectures. For completeness, we review ViT and MViT and introduce notations used in this paper next.…”

Section: Preliminariesmentioning

confidence: 99%

“…Thus the positional embeddings can only indicate the positions within a clip, but not the order across multiple clips. We thus adopt the relative positional embedding used in "the improved MViT" [44], so that memory at different points in time has a different relative distance to the queries.…”

Section: Implementation Detailsmentioning

confidence: 99%

Section: Implementationsmentioning

confidence: 99%

“…Our default MeMViT model is based on MViT-B [22,44] (16 layers) with 16-frame input clips, sampled at a temporal stride of 4 (denoted '16×4' in model specifications). We follow improvements proposed in Li et al [44] due to stronger performance. Following prior work [22][23][24]83], all models in this section are pre-trained on Kinetics-400 [35] unless otherwise stated.…”

Section: Implementationsmentioning

confidence: 99%

See 4 more Smart Citations

MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition

Wu¹,

Li²,

Mangalam³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

While today's video recognition systems parse snapshots or short clips accurately, they cannot connect the dots and reason across a longer range of time yet. Most existing video architectures can only process <5 seconds of a video without hitting the computation or memory bottlenecks.In this paper, we propose a new strategy to overcome this challenge. Instead of trying to process more frames at once like most existing methods, we propose to process videos in an online fashion and cache "memory" at each iteration. Through the memory, the model can reference prior context for long-term modeling, with only a marginal cost. Based on this idea, we build MeMViT, a Memoryaugmented Multiscale Vision Transformer, that has a temporal support 30×longer than existing models with only 4.5% more compute; traditional methods need >3,000% more compute to do the same. On a wide range of settings, the increased temporal support enabled by MeMViT brings large gains in recognition accuracy consistently. MeMViT obtains state-of-the-art results on the AVA, EPIC-Kitchens-100 action classification, and action anticipation datasets. Code and models will be made publicly available.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Preliminariesmentioning

confidence: 99%

Section: Implementation Detailsmentioning

confidence: 99%

Section: Implementationsmentioning

confidence: 99%

Section: Implementationsmentioning

confidence: 99%

See 3 more Smart Citations

MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition

Wu¹,

Li²,

Mangalam³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Frequency Domain Based Learning with Transformer for Underwater Image Restoration

Wang

Sun

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Image deraining is a challenging task that involves restoring degraded images affected by rain streaks. While Convolutional Neural Networks (CNNs) have been commonly used for this task, existing approaches often rely on stacked convolutional basic blocks with limited performance and compromised spatial detail. Furthermore, the limited receptive field of convolutional layers leads to incomplete processing of non-uniform rain streaks. To address these concerns, we propose a novel image deraining network that combines CNNs and transformers. Our network comprises two stages: an encoder-decoder architecture with a triple attention mechanism to capture valuable features and residual dual branch transformer blocks that enhance local information modeling. To address the transformer's lack of local information modeling capability, we introduce convolution in the self-attentive mechanism of the transformer block and feed-forward network. Additionally, we employ a frequency domain contrastive learning method to enhance contrastive sample information, ensuring that the restored image closely resembles the clear image in the frequency domain space, while still retaining a distinction from the rainy image. Extensive quantitative and qualitative experiments demonstrate that our proposed deraining networkoutperforms existing methods on public datasets.

show abstract

Spectrum Attention Mechanism for Time Series Classification

Zhou

Pan

2021

2021 IEEE 10th Data Driven Control and Learning Systems Conference (DDCLS)

View full text Add to dashboard Cite

Time series classification(TSC) has always been an important and challenging research task. With the wide application of deep learning, more and more researchers use deep learning models to solve TSC problems. Since time series always contains a lot of noise, which has a negative impact on network training, people usually filter the original data before training the network. The existing schemes are to treat the filtering and training as two stages, and the design of the filter requires expert experience, which increases the design difficulty of the algorithm and is not universal. We note that the essence of filtering is to filter out the insignificant frequency components and highlight the important ones, which is similar to the attention mechanism. In this paper, we propose an attention mechanism that acts on spectrum (SAM). The network can assign appropriate weights to each frequency component to achieve adaptive filtering. We use L1 regularization to further enhance the frequency screening capability of SAM. We also propose a segmented-SAM (SSAM) to avoid the loss of time domain information caused by using the spectrum of the whole sequence. In which, a tumbling window is introduced to segment the original data. Then SAM is applied to each segment to generate new features. We propose a heuristic strategy to search for the appropriate number of segments. Experimental results show that SSAM can produce better feature representations, make the network converge faster, and improve the robustness and classification accuracy.

show abstract

MViTv2: Improved Multiscale Vision Transformers for Classification and Detection

Cited by 32 publications

References 54 publications

MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition

MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition

Frequency Domain Based Learning with Transformer for Underwater Image Restoration

Spectrum Attention Mechanism for Time Series Classification

Contact Info

Product

Resources

About