Pixel Representation Augmented through Cross-Attention for High-Resolution Remote Sensing Imagery Segmentation

Abstract

Natural imagery segmentation has been transferred to land cover classification in remote sensing imagery with excellent performance. However, two key issues have been overlooked in the transfer process: (1) some objects were easily overwhelmed by the complex backgrounds; (2) interclass information for indistinguishable classes was not fully utilized. The attention mechanism in the transformer is capable of modeling long-range dependencies on each sample for per-pixel context extraction. Notably, per-pixel context from the attention mechanism can aggregate category information. Therefore, we proposed a semantic segmentation method based on pixel representation augmentation. In our method, a simplified feature pyramid was designed to decode the hierarchical pixel features from the backbone, and then decode the category representations into learnable category object embedding queries by cross-attention in the transformer decoder. Finally, pixel representation is augmented by an additional cross-attention in the transformer encoder under the supervision of auxiliary segmentation heads. The results of extensive experiments on the aerial image dataset Potsdam and satellite image dataset Gaofen Image Dataset with 15 categories (GID-15) demonstrate that the cross-attention is effective, and our method achieved the mean intersection over union (mIoU) of 86.2% and 62.5% on the Potsdam test set and GID-15 validation set, respectively. Additionally, we achieved an inference speed of 76 frames per second (FPS) on the Potsdam test dataset, higher than all the state-of-the-art models we tested on the same device.

SABNet: Self-Attention Bilateral Network for Land Cover Classification

“…To further validate the generalisation of the SABNet network, we conducted a series of comparison experiments using the GID-15 dataset with 15 classes and seven other advanced land cover classification methods, namely: BiseNet [22], PSPNet [17], Segformer [41], UNet [16], Deeplabv3+ [26], AMFFNet [48], HPSNet [53] and CAFHNet [54], and calculated three quantitative metrics as well as the number of parameters and computation of the model for the different network experimental results.…”

Deep learning-based semantic segmentation of remote sensing images: a review

“…During the forward propagation process of the CNN, the receptive field increases continuously with the convolution and pooling operations. Multiscale features from channel and spatial can be captured by fusing the features from CNN's different stages (Zheng et al, 2020a;Li Z. et al, 2021;Liu B. et al, 2022;Luo et al, 2022;Wang et al, 2022b;Zhao et al, 2022;Zheng et al, 2022).…”

“…It learned scale-invariant and small objects context information Liu B. et al (2022). designed a method that can efficiently extract different scale features and generate maps, which helps to subdivide objects into small and different sizes Luo et al (2022). extracted categorical object representations from multi-scale pixel features.…”

A Review and Meta-analysis of Semantic Segmentation Models in Land Use/ Land Cover Mapping

“…However, this result can be tackled by hybrid through introduction of encoder-decoder style semantic segmentation models, leverage existing deep learning backbone [70], and explore diverse data settings and parameters in their experimentation [124]. Other methods of structure's enhancement include architectural modifications through the integration of attention mechanisms, transformer architecture, module fusion, and multi-scale feature fusion [125,126]. Example is the SCOCNN framework [127], which addresses the limitation faced by CNN through module integration: A module for semantic segmentation, a module for superpixel optimization, and a module for fusion.…”