Satellite Video Super-Resolution via Multiscale Deformable Convolution Alignment and Temporal Grouping Projection

Xiao, Yi; Su, Xin; Yuan, Qiangqiang; Liu, Denghong; Shen, Huanfeng; Zhang, Liangpei

doi:10.1109/tgrs.2021.3107352

Cited by 115 publications

(51 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the partial CNN in the proposed framework, we set the convolutional filter size as 3 × 3 in all the partial CNN layers (Xiao et al, 2022a). The last partial CNN layer outputs just one feature map and the other partial CNN layers output 64 feature maps (Xiao et al, 2022b).…”

Section: Training and Optimisationmentioning

confidence: 99%

SGD-SM 2.0: an improved seamless global daily soil moisture long-term dataset from 2002 to 2022

Zhang

Yuan²,

Jin³

et al. 2022

Earth Syst. Sci. Data

Self Cite

View full text Add to dashboard Cite

Abstract. The drawbacks of low-coverage rate in global land inevitably exist in satellite-based daily soil moisture products because of the satellite orbit covering scopes and the limitations of soil moisture retrieving models. To solve this issue, Zhang et al. (2021a) generated seamless global daily soil moisture (SGD-SM 1.0) products for the years 2013–2019. Nevertheless, there are still several shortages in SGD-SM 1.0 products, especially in temporal range, sudden extreme weather conditions and sequential time-series information. In this work, we develop an improved seamless global daily soil moisture (SGD-SM 2.0) dataset for the years 2002–2022, to overcome the above-mentioned shortages. The SGD-SM 2.0 dataset uses three sensors, i.e. AMSR-E, AMSR2 and WindSat. Global daily precipitation products are fused into the proposed reconstructing model. We propose an integrated long short-term memory convolutional neural network (LSTM-CNN) to fill the gaps and missing regions in daily soil moisture products. In situ validation and time-series validation testify to the reconstructing accuracy and availability of SGD-SM 2.0 (R: 0.672, RMSE: 0.096, MAE: 0.078). The time-series curves of the improved SGD-SM 2.0 are consistent with the original daily time-series soil moisture and precipitation distribution. Compared with SGD-SM 1.0, the improved SGD-SM 2.0 outperforms on reconstructing accuracy and time-series consistency. The SGD-SM 2.0 products are recorded in https://doi.org/10.5281/zenodo.6041561 (Zhang et al., 2022).

show abstract

Section: Training and Optimisationmentioning

confidence: 99%

SGD-SM 2.0: an improved seamless global daily soil moisture long-term dataset from 2002 to 2022

Zhang

Yuan²,

Jin³

et al. 2022

Earth Syst. Sci. Data

Self Cite

View full text Add to dashboard Cite

show abstract

“…T HE goal of video super-resolution (VSR) is to recover high-resolution (HR) video frames from the lowresolution (LR) ones. This technique has great value in many applications such as satellite imagery [1] and video surveillance [2]. Compared to single-image super-resolution [3]- [6], video super-resolution poses an unsolved challenge of how to fully employ the spatio-temporal dependency since video sequence provides additional temporal information.…”

Section: Introductionmentioning

confidence: 99%

Grouped Spatio-Temporal Alignment Network for Video Super-Resolution

Zhang

2022

IEEE Signal Process. Lett.

View full text Add to dashboard Cite

Video super-resolution based on CNNs has recently achieved significant progress. The existing popular superresolution methods usually impose the optical flow between neighboring frames for temporal alignment. However, estimation of accurate optical flow is hard and expends greater computation.To address this problem, we propose a novel grouped spatiotemporal alignment network (GSTAN) that effectively incorporates spatio-temporal information in a hierarchical way. The input sequence is divided into several groups corresponding to different frame rates. These groups provide complementary information, which is helpful to restore missing textures for the reference frame. Specifically, each group employs deformable 3D convolution to incorporate spatio-temporal information, which avoids artifacts from explicit motion estimation. In addition, a Gated-Dconv information filter is proposed to control information flow to focus on the fine details. Finally, these groups provide complementary information, which is integrated with the intergroup fusion module. Extensive experiments have demonstrated our method achieves state-of-the-art SR performance on several benchmark datasets.

show abstract

“…In recent years, deep neural network (DNN) has made great progress in the field of computer vision. For instance, super-resolution techniques [ 1 , 2 ] help to recover image details to medical disease judgment [ 3 ], and objective recognition tasks [ 4 ] help to segment medical images. However, from the perspective of biological interpretability, a spiking neural network (SNN) is a better choice than a DNN [ 5 , 6 ], which mimics the activity of biological neurons.…”

Section: Introductionmentioning

confidence: 99%

“…When we observe the boundary of a uniformly dark and uniformly bright area, the subjective perception will cause the dark area of the boundary to produce a darker area, and the bright area of the boundary to produce a brighter area. (2) Since the lateral inhibition mechanism can suppress similar information, it can be used as a high-pass lter to suppress the background and low-frequency similar information of the image in space. (3) It has an obvious clustering e ect and can t the subtle discontinuities of the image.…”

Section: Introductionmentioning

confidence: 99%

“…(1) An adaptive lateral inhibition network (LIN) is proposed to suppress interference of background information to detection. (2) Considering that neurons in different brain regions and different types of neurons have different functions and performances, a LIF-based evolving spiking neuron model is proposed. is method provides time constant evolution rules for neurons in different positions in SNN and makes them evolve continuously in the process of network training to ensure the diversity of neurons.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Parallel Spiking Neural Network Based on Adaptive Lateral Inhibition Mechanism for Objective Recognition

Dong

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Spiking neural network (SNN) has attracted extensive attention in the field of machine learning because of its biological interpretability and low power consumption. However, the accuracy of pattern recognition cannot completely surpass deep neural networks (DNNs). The main reason is that the inherent nondifferentiability of spiking neurons makes SNN unable to be trained directly by the gradient descent algorithm, and there is also no unified training algorithm for SNN. Inspired by the biological vision system, this paper proposes a parallel convolution SNN structure combined with an adaptive lateral inhibition mechanism. And, a way of dynamically evolving the time constant with the training of SNN is proposed to ensure the diversity of neurons. This paper verifies the effectiveness of the proposed methods on static datasets and neuromorphic datasets and extends it to the recognition of breast tumors. Experimental results show that the SNN has obvious advantages in dynamical datasets. For breast tumors, it is also an edge-based task, because the edge of a medical image contains the most important information in the image. This kind of information can provide great help for the noninvasive and accurate diagnosis of diseases. The Experimental results show that the proposed method is very close to the recognition results of DNNs on static datasets, and its performance on neuromorphic datasets exceeds that of DNNs.

show abstract

Satellite Video Super-Resolution via Multiscale Deformable Convolution Alignment and Temporal Grouping Projection

Cited by 115 publications

References 60 publications

SGD-SM 2.0: an improved seamless global daily soil moisture long-term dataset from 2002 to 2022

SGD-SM 2.0: an improved seamless global daily soil moisture long-term dataset from 2002 to 2022

Grouped Spatio-Temporal Alignment Network for Video Super-Resolution

A Parallel Spiking Neural Network Based on Adaptive Lateral Inhibition Mechanism for Objective Recognition

Contact Info

Product

Resources

About