Memory-Augmented Transformer for Remote Sensing Image Semantic Segmentation

Abstract

The semantic segmentation of remote sensing images requires distinguishing local regions of different classes and exploiting a uniform global representation of the same-class instances. Such requirements make it necessary for the segmentation methods to extract discriminative local features between different classes and to explore representative features for all instances of a given class. While common deep convolutional neural networks (DCNNs) can effectively focus on local features, they are limited by their receptive field to obtain consistent global information. In this paper, we propose a memory-augmented transformer (MAT) to effectively model both the local and global information. The feature extraction pipeline of the MAT is split into a memory-based global relationship guidance module and a local feature extraction module. The local feature extraction module mainly consists of a transformer, which is used to extract features from the input images. The global relationship guidance module maintains a memory bank for the consistent encoding of the global information. Global guidance is performed by memory interaction. Bidirectional information flow between the global and local branches is conducted by a memory-query module, as well as a memory-update module, respectively. Experiment results on the ISPRS Potsdam and ISPRS Vaihingen datasets demonstrated that our method can perform competitively with state-of-the-art methods.

$\text{A}^{3}$Seg: An Annealing Augmented and Aligned Semantic Segmentation Network for High Spatial Resolution Remote Sensing Images

“…Recently, many transformer-based methods were proposed to learn better feature representation. In [32], Swin Transformers [33] were introduced as the backbone to extract the context information, and a memory-augmented transformer [34] was proposed to model both the local and global feature representation. Beyond bandwise representations in classic transformers, SpectralFormer [35] was developed to learn spectrally local sequence representation.…”

ATFF: Advanced Transformer with Multiscale Contextual Fusion for Medical Image Segmentation

“…TransUNet combines U-Net and Transformer to capture local and global features for medical image segmentation, and achieves excellent segmentation performance [18]. To model local and global context, nn-Former [19] exploited the combination of interleaved convolution and self-attention operations within the encoder and decoder for volumetric medical image segmentation. Furthermore, MISSFormer [20] embedded depth-wise convolution into the transformer block for capturing local and global dependencies.…”

Synthetic Aperture Radar image classification based on constrictive learning with limited data

“…However, this method requires high accuracy of false labels and is prone to receive interference from outliers. In addition, there are also some algorithms for SAR image classification with small samples using techniques in other branches of machine learning, such as generative admissible network [8,9], transfer learning [10,11], recurrent neural network [12], graph neural network [13] etc.…”