Spatio-Temporal Data Fusion for Satellite Images Using Hopfield Neural Network

Abstract

Spatio-temporal data fusion refers to the technique of combining high temporal resolution from coarse satellite images and high spatial resolution from fine satellite images. However, data availability remains a major limitation in algorithm development. Existing spatio-temporal data fusion algorithms require at least one known image pair between the fine and coarse resolution image. However, data which come from two different satellite platforms do not necessarily have an overlap in their overpass times, hence restricting the application of spatio-temporal data fusion. In this paper, a new algorithm named Hopfield Neural Network SPatio-tempOral daTa fusion model (HNN-SPOT) is developed by utilizing the optimization concept in the Hopfield neural network (HNN) for spatio-temporal image fusion. The algorithm derives a synthesized fine resolution image from a coarse spatial resolution satellite image (similar to downscaling), with the use of one fine resolution image taken on an arbitrary date and one coarse image taken on a predicted date. The HNN-SPOT particularly addresses the problem when the fine resolution and coarse resolution images are acquired from different satellite overpass times over the same geographic extent. Both simulated datasets and real datasets over Hong Kong and Australia have been used in the evaluation of HNN-SPOT. Results showed that HNN-SPOT was comparable with an existing fusion algorithm, the spatial and temporal adaptive reflectance fusion model (STARFM). HNN-SPOT assumes consistent spatial structure for the target area between the date of data acquisition and the prediction date. Therefore, it is more applicable to geographical areas with little or no land cover change. It is shown that HNN-SPOT can produce accurate fusion results with >90% of correlation coefficient over consistent land covers. For areas that have undergone land cover changes, HNN-SPOT can still produce a prediction about the outlines and the tone of the features, if they are large enough to be recorded in the coarse resolution image at the prediction date. HNN-SPOT provides a relatively new approach in spatio-temporal data fusion, and further improvements can be made by modifying or adding new goals and constraints in its HNN architecture. Owing to its lower demand for data prerequisites, HNN-SPOT is expected to increase the applicability of fine-scale applications in remote sensing, such as environmental modeling and monitoring.

[Retracted] Target Adaptive Tracking Based on GOTURN Algorithm with Convolutional Neural Network and Data Fusion

“…Since then, literature [6] uses colour histograms as target features to achieve tracking of nonrigid targets, but the problem with this algorithm is that the tracker cannot track the target correctly when the target and the background are similar. Literature [7] uses the method of fusion of colour feature and direction feature to track the target, but this algorithm is not able to track the target well under the condition of scene change. Literature [8] fused colour features and texture features of LBP in a Bayesian framework and used particle filter algorithms for state estimation, which is robust to the deformation and occlusion challenges of the target.…”

Machine Learning Based Analysis of Real-Time Geographical of RS Spatio-Temporal Data

“…The authors in [17] proposed a spatial and temporal reflectance unmixing model (STRUM) algorithm based on the decomposition of mixed pixels and applied it to the normalized difference vegetation index (NDVI) remote sensing image fusion, found that the fusion result of STRUM is better than STARFM. The authors in [18] proposed a spatial and temporal data fusion model (STDFA) based on the temporal change characteristics of pixel reflectivity and the texture characteristics of mediumresolution images method, and the spatial and temporal data fusion model (STDFA) is reconstructed on the NDVI data in Jiangning District, Nanjing. The above spatio-temporal fusion algorithm has been widely used in land surface temperature monitoring [19,20], vegetation change monitoring [21], crop growth monitoring [22,23], etc., but few people in China apply it to flood monitoring, and almost no scholar has explored the applicability of different types of spatiotemporal fusion algorithms in flood monitoring.…”

Multispectral high resolution sensor fusion for smoothing and gap-filling in the cloud

“…The algorithms require building a correspondence between Landsat and MODIS images through learning a dictionary pair resulting in enhancing the spatial resolution of MODIS images to the Landsat spatial resolution. Other approaches for data fusion rely on sophisticated machine learning methods such as convolutional neural networks (CNN), Hopfield neural networks (HNN) and random forests (RF) ( Song et al, 2018 ; Fung et al, 2019 ; Ke et al, 2016 ) to find the relationship between fine and coarse satellite images.…”