Local-Global Temporal Difference Learning for Satellite Video Super-Resolution

Xiao, Yi; Yuan, Qiangqiang; Jiang, Kui; Jin, Xianyu; He, Jiang; Zhang, Liangpei; Lin, Chia-Wen

doi:10.1109/tcsvt.2023.3312321

Cited by 60 publications

(7 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Xiao et al. [33] proposed an efficient and effective temporal compensation method for satellite video SR by exploiting well‐defined temporal differences. They designed a short‐term temporal difference module (S‐TDM), a long‐term temporal difference module (L‐TDM), and a difference compensation unit (DCU) to respectively achieve local information recovery, global compensation, and spatial consistency preservation.…”

Section: Related Workmentioning

confidence: 99%

Self‐attention residual network‐based spatial super‐resolution synthesis for time‐varying volumetric data

Ma,

Ye,

Chen

2024

IET Image Processing

View full text Add to dashboard Cite

In the field of scientific visualization, the upscaling of time‐varying volume is meaningful. It can be used in in situ visualization to help scientists overcome the limitations of I/O speed and storage capacity when analysing and visualizing large‐scale, time‐varying simulation data. This paper proposes self‐attention residual network‐based spatial super‐resolution (SARN‐SSR), a spatial super‐resolution model based on self‐attention residual networks that can generate time‐varying data with temporal coherence. SARN‐SSR consists of two components: a generator and a discriminator. The generator takes the low‐resolution volume sequences as the input and gives the corresponding high‐resolution volume sequences as the output. The discriminator takes both synthesized and real high‐resolution volume sequence as the input and gives a matrix to predict the realness as the output. To verify the validity of SARN‐SSR, four sets of time‐varying volume datasets are applied from scientific simulation. In addition, SARN‐SSR is compared on these datasets, both qualitatively and quantitatively, with two deep learning‐based techniques and one traditional technique. The experimental results show that by using this method, the closest time‐varying data to the ground truth can be obtained.

show abstract

Section: Related Workmentioning

confidence: 99%

Self‐attention residual network‐based spatial super‐resolution synthesis for time‐varying volumetric data

Ma,

Ye,

Chen

2024

IET Image Processing

View full text Add to dashboard Cite

show abstract

“…The most common alternatives to BN are: Group Normalization (GN) [14], Layer Normalization (LN) [15], Instance Normalization (IN) [16] and Batch Renormalization (BRN) [17]. While BN is the most employed normalization method in SOTA algorithms, LN is a fundamental part of the recently proposed Transformer architectures [31] that are becoming widely adopted for solving several learning tasks, such as remote sensing [32], [33] and computer vision [34]. To the best of our knowledge, there are no works benchmarking normalization layers for FL on non-IID data.…”

Section: Related Workmentioning

confidence: 99%

Experimenting With Normalization Layers in Federated Learning on Non-IID Scenarios

Casella,

Esposito,

Sciarappa

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Training Deep Learning (DL) models require large, high-quality datasets, often assembled with data from different institutions. Federated Learning (FL) has been emerging as a method for privacypreserving pooling of datasets employing collaborative training from different institutions by iteratively globally aggregating locally trained models. One critical performance challenge of FL is operating on datasets not independently and identically distributed (non-IID) among the federation participants. Even though this fragility cannot be eliminated, it can be debunked by a suitable optimization of two hyperparameters: layer normalization methods and collaboration frequency selection. In this work, we benchmark five different normalization layers for training Neural Networks (NNs), two families of non-IID data skew, and two datasets. Results show that Batch Normalization, widely employed for centralized DL, is not the best choice for FL, whereas Group and Layer Normalization consistently outperform Batch Normalization, with a performance gain of up to about 15 % in the most challenging non-IID scenario. Similarly, frequent model aggregation decreases convergence speed and mode quality.

show abstract

“…For example, Yi et al [ 15 ] and Li et al [ 16 ] adopted non-local attention. Xiao et al [ 35 ] exploited the temporal difference attention. Wang et al [ 36 ] and Xiao et al [ 37 ] made use of deformable attention.…”

Section: Related Workmentioning

confidence: 99%

Real-Time Video Super-Resolution with Spatio-Temporal Modeling and Redundancy-Aware Inference

Wang,

Liu,

et al. 2023

Sensors

View full text Add to dashboard Cite

Video super-resolution aims to generate high-resolution frames from low-resolution counterparts. It can be regarded as a specialized application of image super-resolution, serving various purposes, such as video display and surveillance. This paper proposes a novel method for real-time video super-resolution. It effectively exploits spatial information by utilizing the capabilities of an image super-resolution model and leverages the temporal information inherent in videos. Specifically, the method incorporates a pre-trained image super-resolution network as its foundational framework, allowing it to leverage existing expertise for super-resolution. A fast temporal information aggregation module is presented to further aggregate temporal cues across frames. By using deformable convolution to align features of neighboring frames, this module takes advantage of inter-frame dependency. In addition, it employs a hierarchical fast spatial offset feature extraction and a channel attention-based temporal fusion. A redundancy-aware inference algorithm is developed to reduce computational redundancy by reusing intermediate features, achieving real-time inferring speed. Extensive experiments on several benchmarks demonstrate that the proposed method can reconstruct satisfactory results with strong quantitative performance and visual qualities. The real-time inferring ability makes it suitable for real-world deployment.

show abstract

Local-Global Temporal Difference Learning for Satellite Video Super-Resolution

Cited by 60 publications

References 67 publications

Self‐attention residual network‐based spatial super‐resolution synthesis for time‐varying volumetric data

Self‐attention residual network‐based spatial super‐resolution synthesis for time‐varying volumetric data

Experimenting With Normalization Layers in Federated Learning on Non-IID Scenarios

Real-Time Video Super-Resolution with Spatio-Temporal Modeling and Redundancy-Aware Inference

Contact Info

Product

Resources

About