Hybrid Attention Fusion Embedded in Transformer for Remote Sensing Image Semantic Segmentation

Abstract

In the context of fast progress in deep learning, convolutional neural networks (CNNs) have been extensively applied to the semantic segmentation of remote sensing images and have achieved significant progress. However, certain limitations exist in capturing global contextual information due to the characteristics of convolutional local properties. Recently, Transformer has become a focus of research in computer vision and has shown great potential in extracting global contextual information, further promoting the development of semantic segmentation tasks. In this paper, we use ResNet50 as an encoder, embed the hybrid attention mechanism into Transformer, and propose a Transformer-based decoder. The Channel-Spatial Transformer Block (CSTB) further aggregates features by integrating the local feature maps extracted by the encoder with their associated global dependencies. At the same time, an adaptive approach is employed to reweight the interdependent channel maps to enhance the feature fusion. The Global Cross-Fusion Module (GCFM) combines the extracted complementary features to obtain more comprehensive semantic information. Extensive comparative experiments were conducted on the ISPRS Potsdam and Vaihingen datasets, where mIoU reached 78.06% and 76.37%, respectively. The outcomes of multiple ablation experiments also validate the effectiveness of the proposed method.

Bidirectional Feature Fusion and Enhanced Alignment Based Multimodal Semantic Segmentation for Remote Sensing Images

“…BEMSeg model was compared with some classic and latest semantic segmentation methods including PSPNet [40], DeepLabV3+ [41], Semantic FPN [33], BANet [42], SwinB-CNN [43], GLOTS [44], HAFNet [45], CLIP-FPN, DenseCLIP [13] and LSeg [15]. CLIP-FPN is a basic multimodal segmentation method that is implemented in this paper.…”