An Experimental Study of the Accuracy and Change Detection Potential of Blending Time Series Remote Sensing Images with Spatiotemporal Fusion

Abstract

Over one hundred spatiotemporal fusion algorithms have been proposed, but convolutional neural networks trained with large amounts of data for spatiotemporal fusion have not shown significant advantages. In addition, no attention has been paid to whether fused images can be used for change detection. These two issues are addressed in this work. A new dataset consisting of nine pairs of images is designed to benchmark the accuracy of neural networks using one-pair spatiotemporal fusion with neural-network-based models. Notably, the size of each image is significantly larger compared to other datasets used to train neural networks. A comprehensive comparison of the radiometric, spectral, and structural losses is made using fourteen fusion algorithms and five datasets to illustrate the differences in the performance of spatiotemporal fusion algorithms with regard to various sensors and image sizes. A change detection experiment is conducted to test if it is feasible to detect changes in specific land covers using the fusion results. The experiment shows that convolutional neural networks can be used for one-pair spatiotemporal fusion if the sizes of individual images are adequately large. It also confirms that the spatiotemporally fused images can be used for change detection in certain scenes.

A Deep Learning-Based Spatio-Temporal NDVI Data Fusion Model

“…Recently, CNN has been modified to produce high accuracy missing data image in spatiotemporal data fusion (Tan et al, 2018). It has opened the door for wide CNN-based spatiotemporal image fusion (Wei et al, 2023). In this context, SRCNN and DCSTFN with three layers of neural networks to downscale the coarse images and to high spatial resolution (Song et al, 2018;Tan et al, 2018).…”

“…Eventually, their applications were extended to the image super-resolution and data fusion domains with direct mapping between input(s) and output (Dong et al, 2015;Liu et al, 2017). Currently, the applications of CNNs on image fusion are being actively explored (Chen et al, 2023). For example, a CNN model was effectively utilized to combine image of a similar scene taken with various central settings to acquire clearness (Liu et al, 2017).…”

“…Then restore the LTHS image at the prediction moment is reduced by deconvolution. Although compared with traditional spatio-temporal data fusion methods, the CNN-based spatio-temporal data fusion model has been proven to be a more potential and efficient way, but its application is still in the preliminary exploration stage and confronts some issues (Wei et al, 2023).…”

A fast adaptive spatio-temporal fusion method to enhanced Fit-FC

“…The procedures and software used for the experiments are as follows: for the OL-Fit-FC method we used the OL software processed by Guo et al [ 43 ]. We also used the eCognition software software for fine classification, and used the multi-resolution segmentation algorithm of the eCognition software to set the smoothing weights and the spectral weights to 0.5 and 0.6, respectively, with a scale value of 150.For FSDAF 2.0, RASDF, Fit-FC, and AL-FF methods were all run on the MA TLAB 2020b platform, and for RASDF, FSDAF 2.0, and Fit-FC parameters were kept the same as the default parameters in their open source codes.The parameters used in the Fit-FC fusion method were carefully selected and determined with reference to previous studies [ 12 , 19 , 44 , 55 ]. For Fit-FC, the RM stage contains 5×5 MODIS pixels with a resolution of 460 m in the moving window, and the SF and RC stages contain Landsat pixels with a resolution of 30 m. The number of similar pixels in Fit-FC is set to 20, and the parameter settings of the AL-FF and Fit-FC methods are basically the same, with the difference that the optimal window is 9, and the AL-FF method is set adaptively, while the Fit-FC method is set manually and empirically to 9.…”

“…For example, the flexible spatio-temporal data fusion (FSDAF) and the reliable adaptive spatio-temporal data fusion method (RASDF). [ 25 ] proposed the Flexible Spatio-Temporal Data Fusion (FSDAF) method, which combines the principles of linear solution mixing models, weight Functions, and spatial interpolation; as a result, FSDAF can recover seasonal and land cover changes, but the algorithm is complex and challenging to capture complex temporal changes [ 44 ]. In addition, RASDF introduces an adaptive local decomposition model that can retrieve substantial temporal variations before filtering and assigning residuals; however, the regional window size is determined empirically in the local solution mixing step, so it is computationally inefficient [ 45 ].…”