Video Super-Resolution Reconstruction Based on Deep Learning and Spatio-Temporal Feature Self-Similarity

Liang, Meiyu; Du, Junping; Li, Linghui; Xue, Zhe; Wang, Xiaoxiao; Kou, Feifei; Wang, Xu

doi:10.1109/tkde.2020.3034261

Cited by 15 publications

(7 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The blue band (from 0.45 µm to 0.49 µm ) of FY4 helps to obtain clear cloud boundary information when drawing cloud cover maps [37]. So we used the NOMChannel01 0.47µm VIS channel of FY4 A data, collected from Sept. 13 As illustrated in Figure 4, the pre-trained FY4 network consists of low-level feature extraction, element-wise summation, and upsampling layers. A collection of features are extracted by the first convolutional layer of the pre-trained FY4 network.…”

Section: A Data Collection and Preprocessingmentioning

confidence: 99%

“…Super-resolution (SR) is a technique for enhancing image spatial resolution by reconstructing high-resolution (HR) images from single or multiple low-resolution (LR) images. It is classified into two categories: classical interpolation methods [8][9][10] and deeplearning-based methods [11][12][13], with the latter subdivided further into single-frame SR methods [14,15] and multi-frame SR methods [16,17]. MFSR constructs HR images by acquiring multiple LR images of the identical scene with the same or distinct sensors [18].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Enhanced Resolution of FY4 Remote Sensing Visible Spectrum Images Utilizing Super-Resolution and Transfer Learning Techniques

Zhang

Wang

et al. 2022

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

Remote sensing images acquired by the FY4 satellite are crucial for regional cloud monitoring and meteorological services. Inspired by the success of deep learning networks in image superresolution, we applied image super-resolution to FY4 visible spectrum (VIS) images. However, training a robust network directly for FY4 VIS image super-resolution remains challenging due to the limited provision of high resolution FY4 sample data. Here, we propose a super-resolution and transfer learning model, FY4-SR-Net. It is composed of pre-training and fine-tuning models. The pre-training model was developed using a deep residual network and a large number of FY4 A 4km and 1km resolution VIS images as the training data. The knowledge derived from 4 km to 1 km resolution images was incorporated into FY4 B 1 km to 0.25 km resolution VIS images. The FY4-SR-Net is finetuned by incorporating limited 1km and 0.25km resolution panchromatic (PAN) images, and then producing 1km superresolution VIS images of the FY4 satellite. Using the one-day FY4 test dataset for qualitative and quantitative evaluations, the FY4-SR-Net outperformed the classic bicubic interpolation approach with a 16.12% reduction in root mean square error (RMSE) and a 2.97% rise in peak signal-to-noise ratio (PSNR) averages. The structural similarity (SSIM) value average increased by 0.0026. This work provides a new precedent for improving the spatial resolution of FY4 series meteorological satellites, which has important scientific significance and application properties.

show abstract

Section: A Data Collection and Preprocessingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Enhanced Resolution of FY4 Remote Sensing Visible Spectrum Images Utilizing Super-Resolution and Transfer Learning Techniques

Zhang

Wang

et al. 2022

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

show abstract

“…Different deep learning based SR-MRI ideas have been proposed for static brain MRI [28,37,38,12,35,39,40,41]. Furthermore, deep learning based methods have additionally been shown to tackle the spatio-temporal trade-off [42], also for dynamic cardiac MR reconstruction [14,15].…”

Section: Related Workmentioning

confidence: 99%

Fine-tuning deep learning model parameters for improved super-resolution of dynamic MRI with prior-knowledge

Sarasaen,

Chatterjee,

Breitkopf

et al. 2021

Preprint

View full text Add to dashboard Cite

Dynamic imaging is a beneficial tool for interventions to assess physiological changes. Nonetheless during dynamic MRI, while achieving a high temporal resolution, the spatial resolution is compromised. To overcome this spatio-temporal trade-off, this research presents a super-resolution (SR) MRI reconstruction with prior knowledge based fine-tuning to maximise spatial information while preserving high temporal resolution of dynamic MRI. An U-Net based network with perceptual loss is trained on a benchmark dataset and fine-tuned using one subject-specific static high resolution MRI as prior knowledge to obtain high resolution dynamic images during the inference stage. 3D dynamic data for three subjects were acquired with different parameters to test the generalisation capabilities of the network. The method was tested for different levels of in-plane undersampling for dynamic MRI. The reconstructed dynamic SR results showed higher similarity with the high resolution ground-truth after fine-tuning. The average SSIM of the lowest resolution experimented during this research (6.25 % of the k-space) before and after fine-tuning were 0.939 ± 0.008 and 0.957 ± 0.006 respectively. This could theoretically result in an acceleration factor of 16, which can potentially be acquired in less than half a second. The proposed approach shows that the super-resolution MRI reconstruction with prior-information can alleviate the spatio-temporal trade-off in dynamic MRI, even for high acceleration factors.

show abstract

“…Recently, temporal information has also been explored in this field. For instance, Kim et al [27] incorporated a deep convolutional neural network and a spatial-temporal feature similarity calculation method to learn the nonlinear correlation mapping between low-resolution and high-resolution video frame patches. Despite progress, these methods of image super-resolution reconstruction can not be directly applied to the more challenging fine-grained traffic flow inference, where coarse-grained traffic flow maps are too rough to maintain the structural information.…”

Section: B Image Super-resolution Reconstructionmentioning

confidence: 99%

Road Network Guided Fine-Grained Urban Traffic Flow Inference

Liu¹,

Liu²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

Accurate inference of fine-grained traffic flow from coarse-grained one is an emerging yet crucial problem, which can help greatly reduce the number of traffic monitoring sensors for cost savings. In this work, we notice that traffic flow has a high correlation with road network, which was either completely ignored or simply treated as an external factor in previous works. To facilitate this problem, we propose a novel Road-Aware Traffic Flow Magnifier (RATFM) that explicitly exploits the prior knowledge of road networks to fully learn the roadaware spatial distribution of fine-grained traffic flow. Specifically, a multi-directional 1D convolutional layer is first introduced to extract the semantic feature of the road network. Subsequently, we incorporate the road network feature and coarse-grained flow feature to regularize the short-range spatial distribution modeling of road-relative traffic flow. Furthermore, we take the road network feature as a query to capture the long-range spatial distribution of traffic flow with a transformer architecture. Benefiting from the road-aware inference mechanism, our method can generate high-quality fine-grained traffic flow maps. Extensive experiments on three real-world datasets show that the proposed RATFM outperforms state-of-the-art models under various scenarios. Our source code and datasets would be released after peer review.

show abstract

Video Super-Resolution Reconstruction Based on Deep Learning and Spatio-Temporal Feature Self-Similarity

Cited by 15 publications

References 49 publications

Enhanced Resolution of FY4 Remote Sensing Visible Spectrum Images Utilizing Super-Resolution and Transfer Learning Techniques

Enhanced Resolution of FY4 Remote Sensing Visible Spectrum Images Utilizing Super-Resolution and Transfer Learning Techniques

Fine-tuning deep learning model parameters for improved super-resolution of dynamic MRI with prior-knowledge

Road Network Guided Fine-Grained Urban Traffic Flow Inference

Contact Info

Product

Resources

About