Global Filter Networks for Image Classification

Abstract

Recent advances in self-attention and pure multi-layer perceptrons (MLP) models for vision have shown great potential in achieving promising performance with fewer inductive biases. These models are generally based on learning interaction among spatial locations from raw data. The complexity of self-attention and MLP grows quadratically as the image size increases, which makes these models hard to scale up when high-resolution features are required. In this paper, we present the Global Filter Network (GFNet), a conceptually simple yet computationally efficient architecture, that learns long-term spatial dependencies in the frequency domain with log-linear complexity. Our architecture replaces the self-attention layer in vision transformers with three key operations: a 2D discrete Fourier transform, an element-wise multiplication between frequency-domain features and learnable global filters, and a 2D inverse Fourier transform. We exhibit favorable accuracy/complexity trade-offs of our models on both ImageNet and downstream tasks. Our results demonstrate that GFNet can be a very competitive alternative to transformer-style models and CNNs in efficiency, generalization ability and robustness. Code is available at https://github.com/raoyongming/GFNet.

MAXIM: Multi-Axis MLP for Image Processing

“…Recent research explorations on Vision Transformers (ViT) [10,23,56] have exemplified their great potential as alternatives to the go-to CNN models. The elegance of ViT [23] has also motivated similar model designs with simpler global operators such as MLP-Mixer [85], gMLP [53], GFNet [74], and FNet [43], to name a few. Despite successful applications to many high-level tasks [4,23,56,83,87,100], the efficacy of these global models on low-level enhancement and restoration problems has not been studied extensively.…”

“…Very recent techniques such as FNet [43] and GFNet [74] demonstrate the simple Fourier Transform can be used as a competitive alternative to either self-attention or MLPs.…”

“…It also allows for significant flexibility and universality. For example, a straightforward replacement of a gMLP with a spatial MLP [85], self-attention [23], or even Fourier Transform [43,74] leads to a family of MAXIM variants (see Sec. 4.3D), all sharing globality and fullyconvolutionality.…”

“…3) offers a scalable way of applying any 1D operators on (highresolution) images, with linear complexity relative to image size while maintaining fully-convolutional. We conducted a pilot study using MAXIM-1S and -2S on SIDD [2] to explore the MAXIM families: MAXIM-FFT, -MLP, -gMLP (modeled in this paper), -SA, where we use the Fourier Transform filter [43,74], spatial MLP [85], gMLP [53], and self-attention [23] on spatial axes using the same multiaxis approach (Fig. 3).…”

Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers

“…More recently, alternatives have been introduced for self-attention that relax the graph assumption for efficient mixing. Instead, they leverage the geometric structures using Fourier transform Rao et al (2021); Lee-Thorp et al (2021). For instance, the Global Filter Networks (GFN) proposes depthwise global convolution for token mixing that enjoys an efficient implementation in the Fourier domain Rao et al (2021).…”