S<sup>2</sup>-MLP: Spatial-Shift MLP Architecture for Vision

Yu, Tan; Liu, Xü; Cai, Yunfeng; Sun, Mingming; Li, Ping

doi:10.1109/wacv51458.2022.00367

Cited by 153 publications

(54 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…They have also been proved to achieve good performance on computer vision tasks. For example, a spatially shifted MLP (S 2 -MLP) (Yu et al, , 2022 takes spatial shifts in four directions and mixes them in a channel-wise manner to gather information from neighboring tokens. Similar to S 2 -MLP, an axial-shifted MLP (AS-MLP) (Lian et al, 2021) changes the spatial shifts in both the horizontal and vertical axes to gather local region information.…”

Section: Related Workmentioning

confidence: 99%

Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs

Zheng¹,

He²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Token-mixing multi-layer perceptron (MLP) models have shown competitive performance in computer vision tasks with a simple architecture and relatively small computational cost. Their success in maintaining computation efficiency is mainly attributed to avoiding the use of self-attention that is often computationally heavy, yet this is at the expense of not being able to mix tokens both globally and locally. In this paper, to exploit both global and local dependencies without self-attention, we present Mix-Shift-MLP (MS-MLP) which makes the size of the local receptive field used for mixing increase in respect to the amount of spatial shifting. In addition to conventional mixing and shifting techniques, MS-MLP mixes both neighboring and distant tokens from fine-to coarse-grained levels and then gathers them via a shifting operation. This directly contributes to the interactions between global and local tokens. Being simple to implement, MS-MLP achieves competitive performance in multiple vision benchmarks. For example, an MS-MLP with 85 million parameters achieves 83.8% top-1 classification accuracy on ImageNet-1K. Moreover, by combining MS-MLP with state-of-the-art Vision Transformers such as the Swin Transformer, we show MS-MLP achieves further improvements on three different model scales, e.g., by 0.5% on ImageNet-1K classification with Swin-B. The code is available at: https://github.com/JegZheng/MS-MLP.

show abstract

Section: Related Workmentioning

confidence: 99%

Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs

Zheng¹,

He²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…ViP [40] and sMLP [34] encode the feature representations along two axial dimensions to improve MLPs' efficiency and capability. Shift [41], ASMLP [29] and S 2 MLP [32] perform spatial information mixing with spatial shift operations along different dimensions. CycleMLP [31], WaveMLP [42] and MorphMLP [33] restrict the spatial information interaction within hand-craft local windows in a deterministic way.…”

Section: Mlp-like Modelsmentioning

confidence: 99%

“…Afterwards, Transformers spring up and make splendid breakthroughs on various vision tasks [13,14,15,16,17,18,19,20,21,22,23,24,25,26,27]. Most recently, the multi-layer perceptrons (MLPs) based architectures [28,29] have regained their light and been demonstrated capable of achieving stunning results on vision tasks [30,28,31,32,29,33,34]. A situation in which these three families of backbone architectures are contending has been formed.…”

Section: Introductionmentioning

confidence: 99%

“…Transformer-based architectures [12,13,14,35,16,15,36,37,17,38,39] perform message passing from other tokens into the query token based on the calculated pairwise attention weights, depending on the affinities between tokens in the embedding space. MLPbased architectures mostly enable information interaction through spatial fully connections across all tokens [28,30,40,34] or across certain tokens selected with hand-crafted rules in a deterministic manner [31,33,41,32,29,42]. However, the fully connection across all tokens makes the network incapable of coping with variable input resolutions, limiting the usage on downstream tasks (e.g., object detection and segmentation).…”

Section: Introductionmentioning

confidence: 99%

“…However, the fully connection across all tokens makes the network incapable of coping with variable input resolutions, limiting the usage on downstream tasks (e.g., object detection and segmentation). Manually designing deterministic rules to restrict the spatial information interaction within fixed regions [31,33,32,29,41,42] eliminates this resolution-fixed constraint but suffers from the lack of adaptability to various visual contents with diverse feature patterns in information interaction.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Active Token Mixer

Wei¹,

Zhang²,

Lan³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper presents ActiveMLP, a general MLP-like backbone for computer vision. The three existing dominant network families, i.e., CNNs, Transformers and MLPs, differ from each other mainly in the ways to fuse contextual information into a given token, leaving the design of more effective token-mixing mechanisms at the core of backbone architecture development. In ActiveMLP, we propose an innovative token-mixer, dubbed Active Token Mixer (ATM), to actively incorporate contextual information from other tokens in the global scope into the given one. This fundamental operator actively predicts where to capture useful contexts and learns how to fuse the captured contexts with the original information of the given token at channel levels. In this way, the spatial range of token-mixing is expanded and the way of token-mixing is reformed. With this design, ActiveMLP is endowed with the merits of global receptive fields and more flexible contentadaptive information fusion. Extensive experiments demonstrate that ActiveMLP is generally applicable and comprehensively surpasses different families of SOTA vision backbones by a clear margin on a broad range of vision tasks, including visual recognition and dense prediction tasks. The code and models will be available at https://github.com/microsoft/ActiveMLP.

show abstract

SRP&PASMLP‐Net: Lightweight skin lesion segmentation network based on structural re‐parameterization and parallel axial shift multilayer perceptron

Wei,

Chen,

Zhao

et al. 2023

Int J Imaging Syst Tech

View full text Add to dashboard Cite

Accurate skin lesion segmentation (SLS) plays an essential role in the computer‐aided diagnosis of skin diseases, for example, melanomas. However, the automated SLS is challenging due to variations in the nature of skin diseases, particularly the ambiguities of boundaries of skin lesion areas (SLAs) and occlusions by hairs. Though many deep learning models, represented by UNet, have been successfully applied to the SLS over the past years, most suffer from inaccurate SLA segmentation with heavy model parameters and slow inferencing speed, hindering their practical applications. To address these challenges, a lightweight SLS network based on structural re‐parameterization and parallel axial shift multilayer perceptron (MLP), called SRP&PASMLP‐Net, is proposed to achieve powerful segmentation performance with fast inference ability. Specifically, a kind of diverse convolution module, the Re‐parameterization Diversity Convolution (RDC) block, is proposed in the shallow stage of the segmentation network to enrich the feature space. The inference is performed by structural re‐parameterization techniques to decouple the structure at the training and inference stages, thus improving the performance of the model without any inference time cost. In addition, a parallel axial shift MLP (PASMLP) module is proposed, which captures local dependencies by axially shifting the channels of the feature map to capture the information flow from different axes and reduce the network parameters and computational complexity effectively. Extensive experimental results show that the proposed SRP&PASMLP‐Net can provide a 4‐fold improvement in the inference speed and a 17‐fold reduction in model parameters compared with the benchmark network, UNet. It outperforms the state‐of‐the‐art methods on two public datasets, that is, ISIC2018 and PH2, for the SLS.

show abstract

S²-MLP: Spatial-Shift MLP Architecture for Vision

Cited by 153 publications

References 18 publications

Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs

Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs

Active Token Mixer

SRP&PASMLP‐Net: Lightweight skin lesion segmentation network based on structural re‐parameterization and parallel axial shift multilayer perceptron

Contact Info

Product

Resources

About

S2-MLP: Spatial-Shift MLP Architecture for Vision

Cited by 153 publications

References 18 publications

Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs

Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs

Active Token Mixer

SRP&PASMLP‐Net: Lightweight skin lesion segmentation network based on structural re‐parameterization and parallel axial shift multilayer perceptron

Contact Info

Product

Resources

About

S²-MLP: Spatial-Shift MLP Architecture for Vision