Inception Transformer

Abstract

Recent studies show that Transformer has strong capability of building long-range dependencies, yet is incompetent in capturing high frequencies that predominantly convey local information. To tackle this issue, we present a novel and general-purpose Inception Transformer, or iFormer for short, that effectively learns comprehensive features with both high-and low-frequency information in visual data. Specifically, we design an Inception mixer to explicitly graft the advantages of convolution and max-pooling for capturing the high-frequency information to Transformers. Different from recent hybrid frameworks, the Inception mixer brings greater efficiency through a channel splitting mechanism to adopt parallel convolution/max-pooling path and self-attention path as high-and low-frequency mixers, while having the flexibility to model discriminative information scattered within a wide frequency range. Considering that bottom layers play more roles in capturing high-frequency details while top layers more in modeling low-frequency global information, we further introduce a frequency ramp structure, i.e., gradually decreasing the dimensions fed to the high-frequency mixer and increasing those to the low-frequency mixer, which can effectively trade-off high-and lowfrequency components across different layers. We benchmark the iFormer on a series of vision tasks, and showcase that it achieves impressive performance on image classification, COCO detection and ADE20K segmentation. For example, our iFormer-S hits the top-1 accuracy of 83.4% on ImageNet-1K, much higher than DeiT-S by 3.6%, and even slightly better than much bigger model Swin-B (83.3%) with only 1/4 parameters and 1/3 FLOPs. Code and models will be released at https://github.com/sail-sg/iFormer. * Equal contribution. Weihao Yu did this work during an internship at Sea AI Lab.Preprint. Under review.

E-Branchformer: Branchformer with Enhanced merging for speech recognition

“…Recently, Inception Transformer [45] which has three branches (average pooling, convolution, and self-attention) fused with a depthwise convolution achieves impressive performance on several vision tasks. Our E-Branchformer shares a similar spirit of combing local and global information both sequentially and in parallel.…”

“…Presumably, using nearby information can improve the merge process. Similar to Inception-Transformer [45], we employ a depth-wise convolution to add the spatial information exchanging (as described in Figure 3c). Formally, the outputs from the global Y G and the local Y L branch are merged:…”

Parallel matters: Efficient polyp segmentation with parallel structured feature augmentation modules

“…Xie et al [20] proposed a framework that outputs multi-scale features by a hierarchical Transformer encoder and saves on attentional computation. Recently, Inception Transformer [12] adds an additional Inception module inside a Transformer for extracting high-frequency representation, so that more strong performances of Transformer can be obtained. We, however, aim at combining different Transformers with different reception fields in the Inception style for robust feature abstraction.…”

“…Interestingly, Transformer [5] can capture the long-range relations. Besides, the parallel structures in the convolutional neural networks (CNN)-based studies, Inception [10] and its variants [11][12][13][14] have been demonstrated to be very effective with rich scales.…”

ITSRN++: Stronger and Better Implicit Transformer Network for Continuous Screen Content Image Super-Resolution

“…For the convolution branch, different from [53], [54], [55], which perform convolution with the input features, we instead extract the convolution features from the value V , which is not partitioned into windows. In this way, the convolution layer can explore the correlations among neighboring windows, which further enhances the correlations of tokens along the window borders.…”