A Deep Multitask Learning Framework Coupling Semantic Segmentation and Fully Convolutional LSTM Networks for Urban Change Detection

Abstract

In this paper, we present a deep multi-task learning framework able to couple semantic segmentation and change detection using fully convolutional long short-term memory (LSTM) networks. In particular, we present a UNet-like architecture (L-UNet) which models the temporal relationship of spatial feature representations using integrated fully convolutional LSTM blocks on top of every encoding level. In this way, the network is able to capture the temporal relationship of spatial feature vectors in all encoding levels without the need to downsample or flatten them, forming an end-to-end trainable framework. Moreover, we further enrich the L-UNet architecture with an additional decoding branch that performs semantic segmentation on the available semantic categories that are presented in the different input dates, forming a multi-task framework. Different loss quantities are also defined and combined together in a circular way to boost the overall performance. The developed methodology has been evaluated on three different datasets, i.e, the challenging bi-temporal high-resolution ONERA Satellite Change Detection (OSCD) Sentinel-2 dataset, the very high-resolution multitemporal dataset of the East Prefecture of Attica, Greece, and lastly, the multitemporal very high-resolution SpaceNet7 dataset. Promising quantitative and qualitative results demonstrated that the synergy among the tasks can boost up the achieved performances. In particular, the proposed multi-task framework contributed to a significant decrease of false positive detections, with F1 rate outperforming other state of the art methods by at least 2.1% and 4.9% in the Attica VHR and SpaceNet7 dataset case respectively. Our models and code can be found at: https://github.com/mpapadomanolaki/multi-task-L-UNet

Deep Learning-Based Change Detection in Remote Sensing Images: A Review

“…Papadomanolaki et al [167] proposed a unique model using fully convolutional LSTM networks and presented a U-Net-like architecture (LU-Net) that models the temporal relationship of spatial feature representations by layering integrated fully convolutional LSTM blocks on top of each encoding level and with an additional decoding branch that performs semantic segmentation on the available semantic categories presented in the various input dates, resulting in a multitask framework.…”

ForkNet: Strong Semantic Feature Representation and Subregion Supervision for Accurate Remote Sensing Change Detection

“…Long short-term memory (LSTM) and generative adversarial network (GAN) are also integrated into change detection methods. Papadomanolaki et al [23] and Sun et al [24] both proposed to combine LSTM and convolution for change detection in the U-Net architecture, which can model the temporal relationships between spatial feature representations. Lebedev et al [15] modified the "pix2pix" architecture of GAN to perform the RSCD task, which shows the potential of GAN networks for the RSCD task, although the network is not effective in detecting small-sized objects.…”

“…Lebedev et al [15] modified the "pix2pix" architecture of GAN to perform the RSCD task, which shows the potential of GAN networks for the RSCD task, although the network is not effective in detecting small-sized objects. The change detection method integrating LSTM has a great improvement for multitemporal remote sensing image change detection because the temporal patterns in multitemporal remote sensing image data can be captured [23]. However, for bitemporal remote sensing image change detection tasks, whether LSTM can effectively model the temporal patterns in them still needs to be further explored, especially when bitemporal images span several years containing several seasons.…”

MAFF-Net: Multi-Attention Guided Feature Fusion Network for Change Detection in Remote Sensing Images

“…Remote sensing image change detection (CD) uses two or more remote sensing images of the same area at different times to compare and analyze the atmospheric, spectral, and sensor information through artificial intelligence or mathematical statistics to obtain the change information of the area [ 1 , 2 ]. CD is an important research direction in the field of remote sensing and plays a great role in many fields such as land planning, urban expansion [ 3 , 4 ], environmental monitoring [ 5 , 6 , 7 ], and disaster assessment [ 8 ] as a key technology for monitoring surface conditions.…”