SMNet: Symmetric Multi-Task Network for Semantic Change Detection in Remote Sensing Images Based on CNN and Transformer

Abstract

Deep learning has achieved great success in remote sensing image change detection (CD). However, most methods focus only on the changed regions of images and cannot accurately identify their detailed semantic categories. In addition, most CD methods using convolutional neural networks (CNN) have difficulty capturing sufficient global information from images. To address the above issues, we propose a novel symmetric multi-task network (SMNet) that integrates global and local information for semantic change detection (SCD) in this paper. Specifically, we employ a hybrid unit consisting of pre-activated residual blocks (PR) and transformation blocks (TB) to construct the (PRTB) backbone, which obtains more abundant semantic features with local and global information from bi-temporal images. To accurately capture fine-grained changes, the multi-content fusion module (MCFM) is introduced, which effectively enhances change features by distinguishing foreground and background information in complex scenes. In the meantime, the multi-task prediction branches are adopted, and the multi-task loss function is used to jointly supervise model training to improve the performance of the network. Extensive experimental results on the challenging SECOND dataset demonstrate that our SMNet obtains 71.95% and 20.29% at mean Intersection over Union (mIoU) and Separated Kappa coefficient (Sek), respectively, which proves the effectiveness and superiority of the proposed method.

A hybrid U‐shaped and transformer network for change detection in high‐resolution remote sensing images

“…To simplify the transformer block, Lin et al [44] presented the swin transformer framework by incorporating hierarchical, locality, and shift invariance priors, which achieves better performance in various tasks. Inspired by transformer, some CD methods based on transformer and CNN are proposed [45][46][47][48][49][50]. Shi et al [51] designed the bi-temporal image transformer (BIT), a method based on transformer to address CD challenges.…”

“…[44] presented the swin transformer framework by incorporating hierarchical, locality, and shift invariance priors, which achieves better performance in various tasks. Inspired by transformer, some CD methods based on transformer and CNN are proposed [45–50]. Shi et al.…”

A panoramic driving perception fusion algorithm based on multi-task learning

“…In order to enhance the performance and practicality of panoramic driving perception systems, numerous studies have been dedicated to designing more efficient and accurate multitask learning networks. In recent research, multi-task learning networks such as DeMT [10] (Deformable Mixer Transformer), SMNet [11] (Symmetric Multi-task Network), YOLOP [12], HybridNets [13], and YOLOPv2 [14] have gradually integrated single tasks into multitasks and processed them simultaneously to impove performance. However, these multi-task learning networks still face certain challenges in current low-cost autonomous driving applications.…”

SIGNet: A Siamese Graph Convolutional Network for Multi-Class Urban Change Detection

“…To address this issue, the latest research integrates global and local information through the combination of CNN and transformers. Niu et al [69] introduced the multi-content fusion module to facilitate the extraction of change features in complex contexts by fusing foreground, background, and global information. Cui et al [70] explored the relationship between semantic segmentation and BCD and further improved detection performance by utilizing the correlation between the two subtasks.…”