MetaFormer Is Actually What You Need for Vision

Yu, Weihao; Luo, Mi; Zhou, Pan; Si, Chenyang; Zhou, Yichen; Wang, Xinchao; Feng, Jiashi; Yan, Shuicheng

doi:10.48550/arxiv.2111.11418

Cited by 32 publications

(59 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These differences in mechanism lead to significant improvement of efficiency and performance as well. Another closely related work is Poolformer [74] which uses a pooling to summarize the local context and a simple subtraction to adjust the individual inputs. Though achieving decent efficiency for its simplicity, Poolformer lags behind popular vision transformers like Swin on performance.…”

Section: Related Workmentioning

confidence: 99%

“…( 4) is motivated by its desirable properties. Compared to pooling [74,29], depth-wise convolution is learnable and structure-aware. In contrast to regular convolution, it is channel-wise and thus computationally much cheaper.…”

Section: Context Aggregation Via Mmentioning

confidence: 99%

“…-Pooling Aggregator. It replaces the depth-wise convolution module with average pooling, and is similar to MetaFormer [74] in terms of token aggregation. Average pooling has slightly lower complexity but leads to a significant drop of accuracy (1.8%).…”

Section: Network Inspectionmentioning

confidence: 99%

See 2 more Smart Citations

Focal Modulation Networks

Yang¹,

Li²,

Gao³

2022

Preprint

View full text Add to dashboard Cite

In this work, we propose focal modulation network (FocalNet in short), where self-attention (SA) is completely replaced by a focal modulation module that is more effective and efficient for modeling token interactions. Focal modulation comprises three components: (i) hierarchical contextualization, implemented using a stack of depth-wise convolutional layers, to encode visual contexts from short to long ranges at different granularity levels, (ii) gated aggregation to selectively aggregate context features for each visual token (query) based on its content, and (iii) modulation or element-wise affine transformation to fuse the aggregated features into the query vector. Extensive experiments show that FocalNets outperform the state-of-the-art SA counterparts (e.g., Swin Transformers) with similar time and memory cost on the tasks of image classification, object detection, and semantic segmentation. Specifically, our FocalNets with tiny and base size achieve 82.3% and 83.9% top-1 accuracy on ImageNet-1K. After pretrained on ImageNet-22K, it attains 86.5% and 87.3% top-1 accuracy when finetuned with resolution 224 2 and 384 2 , respectively. FocalNets exhibit remarkable superiority when transferred to downstream tasks. For object detection with Mask R-CNN, our FocalNet base trained with 1× already surpasses Swin trained with 3× schedule (49.0 v.s. 48.5). For semantic segmentation with UperNet, FocalNet base evaluated at single-scale outperforms Swin evaluated at multi-scale (50.5 v.s. 49.7). These results render focal modulation a favorable alternative to SA for effective and efficient visual modeling in real-world applications.Code is available at: https://github.com/microsoft/FocalNet.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Context Aggregation Via Mmentioning

confidence: 99%

See 1 more Smart Citation

Focal Modulation Networks

Yang¹,

Li²,

Gao³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We evaluate the effectiveness of Mugs through ViT [23,39] and thus will take ViT as an example to introduce Mugs. This is because 1) with similar model size, ViT shows better performance than CNN [49,39,28,58]; 2) ViT shows great potential for unifying the vision and language models [28,2].…”

Section: Multi-granular Self-supervised Learningmentioning

confidence: 99%

Mugs: A Multi-Granular Self-Supervised Learning Framework

Zhou¹,

Zhou²,

Si³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In self-supervised learning, multi-granular features are heavily desired though rarely investigated, as different downstream tasks (e.g., general and fine-grained classification) often require different or multigranular features, e.g. fine-or coarse-grained one or their mixture. In this work, for the first time, we propose an effective MUlti-Granular Selfsupervised learning (Mugs) framework to explicitly learn multi-granular visual features. Mugs has three complementary granular supervisions: 1) an instance discrimination supervision (IDS), 2) a novel local-group discrimination supervision (LGDS), and 3) a group discrimination supervision (GDS). IDS distinguishes different instances to learn instancelevel fine-grained features.LGDS aggregates features of an image and its neighbors into a local-group feature, and pulls local-group features from different crops of the same image together and push them away for others. It provides complementary instance supervision to IDS via an extra alignment on local neighbors, and scatters different local-groups separately to increase discriminability. Accordingly, it helps learn high-level fine-grained features at a local-group level. Finally, to prevent similar local-groups from being scattered randomly or far away, GDS brings similar samples close and thus pulls similar local-groups together, capturing coarse-grained features at a (semantic) group level. Consequently, Mugs can capture three granular features that often enjoy higher generality on diverse downstream tasks over single-granular features, e.g. instancelevel fine-grained features in contrastive learning. By only pretraining on ImageNet-1K, Mugs sets new SoTA linear probing accuracy 82.1% on ImageNet-1K and improves previous SoTA by 1.1%. It also surpasses SoTAs on other tasks, e.g. transfer learning, detection and segmentation. Codes and models are available at https://github.com/sail-sg/mugs.

show abstract

“…Transformer models [48], [49], [50], [51] have recently achieved excellent performance on a wide range of language and computer vision tasks, e.g., machine translation [52], image recognition [53], video understanding [54], visual question answering [55], etc. Generally, the success of Transformer can be attributed to its selfsupervision and self-attention [50].…”

Section: Transformer Modelmentioning

confidence: 99%

TransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learning

Chen¹,

Hong²,

Xie³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Zero-shot learning (ZSL) tackles the novel class recognition problem by transferring semantic knowledge from seen classes to unseen ones. Semantic knowledge is typically represented by attribute descriptions shared between different classes, which act as strong priors for localizing object attributes that represent discriminative region features, enabling significant and sufficient visual-semantic interaction for advancing ZSL. Existing attention-based models have struggled to learn inferior region features in a single image by solely using unidirectional attention, which ignore the transferability and discriminative attribute localization of visual features. In this paper, we propose a cross attribute-guided Transformer network, termed TransZero++, to refine visual features and learn accurate attribute localization for semantic-augmented visual embedding representations in ZSL. TransZero++ consists of an attribute→visual Transformer sub-net (AVT) and a visual→attribute Transformer sub-net (VAT). Specifically, AVT first takes a feature augmentation encoder to alleviate the cross-dataset bias between ImageNet and ZSL benchmarks, and improves the transferability of visual features by reducing the entangled relative geometry relationships among region features. Then, an attribute→visual decoder is employed to localize the image regions most relevant to each attribute in a given image for attribute-based visual feature representations. Analogously, VAT uses the similar feature augmentation encoder to refine the visual features, which are further applied in visual→attribute decoder to learn visual-based attribute features. By further introducing feature-level and prediction-level semantical collaborative losses, the two attribute-guided transformers teach each other to learn semantic-augmented visual embeddings via semantical collaborative learning. Finally, the semantic-augmented visual embeddings learned by AVT and VAT are fused to conduct desirable visual-semantic interaction cooperated with semantic vectors for ZSL classification. Extensive experiments show that TransZero++ achieves the new state-of-the-art results on three golden and challenging ZSL benchmarks. The codes are available at: https://github.com/shiming-chen/TransZero_pp.

show abstract

MetaFormer Is Actually What You Need for Vision

Cited by 32 publications

References 48 publications

Focal Modulation Networks

Focal Modulation Networks

Mugs: A Multi-Granular Self-Supervised Learning Framework

TransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learning

Contact Info

Product

Resources

About