An Enhanced Deep Convolutional Model for Spatiotemporal Image Fusion

Abstract

Earth observation data with high spatiotemporal resolution are critical for dynamic monitoring and prediction in geoscience applications, however, due to some technique and budget limitations, it is not easy to acquire satellite images with both high spatial and high temporal resolutions. Spatiotemporal image fusion techniques provide a feasible and economical solution for generating dense-time data with high spatial resolution, pushing the limits of current satellite observation systems. Among existing various fusion algorithms, deeplearningbased models reveal a promising prospect with higher accuracy and robustness. This paper refined and improved the existing deep convolutional spatiotemporal fusion network (DCSTFN) to further boost model prediction accuracy and enhance image quality. The contributions of this paper are twofold. First, the fusion result is improved considerably with brand-new network architecture and a novel compound loss function. Experiments conducted in two different areas demonstrate these improvements by comparing them with existing algorithms. The enhanced DCSTFN model shows superior performance with higher accuracy, vision quality, and robustness. Second, the advantages and disadvantages of existing deeplearningbased spatiotemporal fusion models are comparatively discussed and a network design guide for spatiotemporal fusion is provided as a reference for future research. Those comparisons and guidelines are summarized based on numbers of actual experiments and have promising potentials to be applied for other image sources with customized spatiotemporal fusion networks.

A Hybrid Deep Learning-Based Spatiotemporal Fusion Method for Combining Satellite Images with Different Resolutions

“…Differently from existing DL-based STF methods that need at least two pairs of finecoarse images [42,48,[53][54][55], HDLSFM requires minimal input data, i.e., a known fine ( ) and a coarse ( ) image at a base date ( ), and a coarse image ( ) at a prediction date ( ), to generate a robust target fine image ( ) containing both PC and LC information. This is achieved through a hybrid framework consisting of a nonlinear DL-based relative radiometric normalization that alleviates radiation differences, a DL-based SR algorithm for LC prediction, and linear-based fusion for PC prediction.…”

“…In order to convert the spatial resolution of image ( ) from low to high, spatial interpolation, such as thin plate spline interpolation, has been widely utilized in STF methods [12,44]. In this paper, the DL-based SR method is used to retain the complementary LC information, which is considered to be more beneficial for accurate LC prediction [36], and underpins a higher generalization ability because of the introduction of prior information [53,55]. According to the core of the DL-based SR method, the nonlinear mapping between and is formulated as:…”

Applications of Deep Learning-Based Super-Resolution for Sea Surface Temperature Reconstruction

“…The CNN, one of the most popular networks in deep learning, has been widely employed in many remote sensing research fields, such as image classification [32,33] and image fusion [34][35][36][37][38]. Three important architecture ideas including the local receptive field, weight sharing and subsampling, enable CNNs to achieve shift, scale and distortion invariant properties [34].…”

A Novel Deep Learning-Based Spatiotemporal Fusion Method for Combining Satellite Images with Different Resolutions Using a Two-Stream Convolutional Neural Network

“…Additionally, MSE is considered to generate the overly smooth effect, so it may also be the reason that the visual effect of the DL-SDFM is slightly inferior to that of STFDCNN. Therefore, future work should improve the visual effect further by employing a more appropriate loss function to replace the MSE or using a combination of multiple loss functions [49,50].…”