RSTT: Real-time Spatial Temporal Transformer for Space-Time Video Super-Resolution

Geng, Zhicheng; Liang, Luming; Ding, Tianyu; Zharkov, Ilya

doi:10.1109/cvpr52688.2022.01692

Cited by 64 publications

(23 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The entire compression pipeline is depicted in Stage (I) of Figure 2. We remark that a pre-trained M 0 is not essential for the purpose of compression since problem (2) is sufficient for a one-shot training/pruning, but M 0 allows us to ensure the compressed model performs competitively.…”

Section: General Methodologymentioning

confidence: 99%

“…V IDEO frame interpolation is a low-level computer vision task that involves creating interim (non-existent) frames between actual frames in a sequence to greatly improve the temporal resolution. It plays an important role in many applications, including frame rate up-conversion [1], [2], slow-motion generation [3], and novel view synthesis [4], [5]. Though fundamental, the problem is challenging in that the complex motion, occlusion and feature variation in real world videos are difficult to estimate and predict in a transparent way.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

CDFI: Compression-Driven Network Design for Frame Interpolation

Ding

Liang

Zhu

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

View full text Add to dashboard Cite

DNN-based frame interpolation, which generates intermediate frames from two consecutive frames, is often dependent on model architectures with a large number of features, preventing their deployment on systems with limited resources, such as mobile devices. We present a compression-driven network design for frame interpolation that leverages model pruning through sparsity-inducing optimization to greatly reduce the model size while attaining higher performance. Concretely, we begin by compressing the recently proposed AdaCoF model and demonstrating that a 10× compressed AdaCoF performs similarly to its original counterpart, where different strategies for using layerwise sparsity information as a guide are comprehensively investigated under a variety of hyperparameter settings. We then enhance this compressed model by introducing a multi-resolution warping module, which improves visual consistency with multi-level details. As a result, we achieve a considerable performance gain with a quarter of the size of the original AdaCoF. In addition, our model performs favorably against other state-of-the-art approaches on a wide variety of datasets. We note that the suggested compression-driven framework is generic and can be easily transferred to other DNN-based frame interpolation algorithms. The source code is available at https://github.com/tding1/CDFI.

show abstract

Section: General Methodologymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

CDFI: Compression-Driven Network Design for Frame Interpolation

Ding

Liang

Zhu

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

View full text Add to dashboard Cite

show abstract

“…To interpolate an arbitrary number of intermediate frames for STVSR, Xu et al [31] proposed a temporal modulation network, which is achieved by temporal modulation block under the deformable convolution framework for controllable feature interpolation. Recently, Geng et al [32] proposed a single spatial temporal Transformer architecture that incorporates the temporal interpolation and spatial super-resolution modules for the STVSR task. To sufficiently aggregate the spatio-temporal information, Hu et al [33] proposed an efficient recurrent network with bidirectional interactive propagation module for STVSR task, where only one alignment and fusion are required.…”

Section: Two-stage and One-stage Space-time Video Super-resolutionmentioning

confidence: 99%

“…For the four super-resolution methods, Bicubic and RCAN [47] are single-image super-resolution methods, RBPN [15] and EDVR [21] are recent VSR methods. In addition, the one-stage STVSR methods used for comparison include Zooming SlowMo [30] and RSTT [32].…”

Section: Evaluation On Space-time Video Super-resolutionmentioning

confidence: 99%

Non‐local neural networks combined with local importance‐based pooling for space‐time video super‐resolution

Zhang

Lei

Zhang

et al. 2022

IET Image Processing

View full text Add to dashboard Cite

Compared with convolutional operation, non‐local operation can directly capture long‐range dependencies and thus has a larger receptive field. However, the computation and memory consumption of non‐local operation is much higher than convolutional operation, so it cannot be used repeatedly as a general component directly. In this paper, in order to balance the accuracy and computational complexity of non‐local enhancement, the non‐local operation is simplified based on local importance‐based pooling, which can dynamically extract discriminative features during the down‐sampling process by learning adaptive weights. Such simplified non‐local enhancement is able to prevent unacceptable computational consumption caused by directly processing the entire feature maps containing a large number of features. In order to verify the effectiveness of the proposed method, 2D and 3D feature extraction blocks are constructed based on the simplified non‐local operations, and they are stacked as feature extraction networks for space‐time video super‐resolution task, which aims to increase resolution in both time and space simultaneously. Extensive experiments demonstrate that the proposed simplified non‐local networks can effectively improve the performance of space‐time video super‐resolution task both quantitatively and qualitatively.

show abstract

“…Based on the swin transformer and convolution, a spatiotemporal model is constructed in this study to perform the bias correction task and the temporal downscaling task, as shown in Figure 2. The spatio-temporal model based on the swin transformer has been designed and applied to video superresolution tasks and action recognition tasks (Geng et al, 2022;Liu et al, 2022). The entire framework consists of an encoder, a decoder, and a query builder.…”

Section: Modelmentioning

confidence: 99%

Spatiotemporal model based on transformer for bias correction and temporal downscaling of forecasts

Guan

Xiang

et al. 2022

Front. Environ. Sci.

View full text Add to dashboard Cite

Numerical weather prediction (NWP) provides the future state of the atmosphere and is a major tool for weather forecasting. However, NWP has inevitable errors and requires bias correction to obtain more accurate forecasts. NWP is based on discrete numerical calculations, which inevitably result in a loss in resolution, and downscaling provides important support for obtaining detailed weather forecasts. In this paper, based on the spatio-temporal modeling approach, the Spatio-Temporal Transformer U-Net (ST-UNet) is constructed based on the U-net framework using the swin transformer and convolution to perform bias correction and temporal downscaling. The encoder part extracts features from the multi-time forecasts, and the decoder part uses the features from the encoder part and the constructed query vector for feature reconstruction. Besides, the query builder block generates different query vectors to accomplish different tasks. Multi-time bias correction was conducted for the 2-m temperature and the 10-m wind component. The results showed that the deep learning model significantly outperformed the anomaly numerical correction with observations, and ST-UNet also outperformed the U-Net model for single-time bias correction and the 3-dimensional U-Net (3D-UNet) model for multi-time bias correction. Forecasts from ST-UNet obtained the smallest root mean square error and the largest accuracy and correlation coefficient on both the 2-m temperature and 10-m wind component experiments. Meanwhile, temporal downscaling was performed to obtain hourly forecasts based on ST-UNet, which increased the temporal resolution and reduced the root mean square error by 0.78 compared to the original forecasts. Therefore, our proposed model can be applied to both bias correction and temporal downscaling tasks and achieve good accuracy.

show abstract

RSTT: Real-time Spatial Temporal Transformer for Space-Time Video Super-Resolution

Cited by 64 publications

References 40 publications

CDFI: Compression-Driven Network Design for Frame Interpolation

CDFI: Compression-Driven Network Design for Frame Interpolation

Non‐local neural networks combined with local importance‐based pooling for space‐time video super‐resolution

Spatiotemporal model based on transformer for bias correction and temporal downscaling of forecasts

Contact Info

Product

Resources

About