Space-Time-Aware Multi-Resolution Video Enhancement

Haris, Muhammad; Shakhnarovich, Greg; Ukita, Norimichi

doi:10.1109/cvpr42600.2020.00293

Cited by 113 publications

(111 citation statements)

References 46 publications

Supporting

Mentioning

111

Contrasting

Order By: Relevance

“…Very recently, [21] proposed a multi-scale temporal loss. [14] concatenated LR images and pre-computed optical flow for intermediate feature estimation and refinement. [20] used optical flow to explicitly warp the features.…”

Section: Space-time Video Super-resolutionmentioning

confidence: 99%

Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution

Xiang

Tian

Zhang

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

185

210

View full text Add to dashboard Cite

In this paper, we address the space-time video super-resolution, which aims at generating a high-resolution (HR) slow-motion video from a low-resolution (LR) and low frame rate (LFR) video sequence. A naive method is to decompose it into two sub-tasks: video frame interpolation (VFI) and video super-resolution (VSR). Nevertheless, temporal interpolation and spatial upscaling are intra-related in this problem. Two-stage approaches cannot fully make use of this natural property. Besides, state-of-the-art VFI or VSR deep networks usually have a large frame reconstruction module in order to obtain high-quality photo-realistic video frames, which makes the two-stage approaches have large models and thus be relatively time-consuming. To overcome the issues, we present a one-stage space-time video super-resolution framework, which can directly reconstruct an HR slow-motion video sequence from an input LR and LFR video. Instead of reconstructing missing LR intermediate frames as VFI models do, we temporally interpolate LR frame features of the missing LR frames capturing local temporal contexts by a feature temporal interpolation module. Extensive experiments on widely used benchmarks demonstrate that the proposed framework not only achieves better qualitative and quantitative performance on both clean and noisy LR frames but also is several times faster than recent state-of-the-art two-stage networks. The source code is released in https:// github.com/ Mukosame/ Zooming-Slow-Mo-CVPR-2020.

show abstract

Section: Space-time Video Super-resolutionmentioning

confidence: 99%

Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution

Xiang

Tian

Zhang

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

185

210

View full text Add to dashboard Cite

show abstract

“…By making use of mature optical flow packages, both neighborhood frames and optical flow are fed into the network for joint spatial and temporal feature extraction. For example, [51]- [53] propose to directly input neighborhood frames and optical flows for super-resolution via 2D convolution. [54], [55] propose deformable convolution operations for flexible subpixel motion estimation.…”

Section: Related Workmentioning

confidence: 99%

Efficient Video Super-Resolution via Hierarchical Temporal Residual Networks

2021

View full text Add to dashboard Cite

Super-Resolving (SR) video is more challenging compared with image super-resolution because of the demanding computation time. To enlarge a low-resolution video, the temporal relationship among frames must be fully exploited. We can model video SR as a multi-frame SR problem and use deep learning methods to estimate the spatial and temporal information. This paper proposes a lighter residual network, based on a multi-stage back projection for multi-frame SR. We improve the back projection based residual block by adding weights for adaptive feature tuning, and add global & local connections to explore deeper feature representation. We jointly learn spatial-temporal feature maps by using the proposed Spatial Convolution Packing scheme as an attention mechanism to extract more information from both spatial and temporal domains. Different from others, our proposed network can input multiple low-resolution frames to obtain multiple super-resolved frames simultaneously. We can then further improve the video SR quality by self-ensemble enhancement to meet videos with different motions and distortions. Results of much experimental work show that our proposed approaches give large improvement over other state-of-the-art video SR methods. Compared to recent CNN based video SR works, our approaches can save, up to 60% computation time and achieve 0.6 dB PSNR improvement.

show abstract

“…The team's base network for x16 video SR is STARnet [13] shown in Fig.10. With the idea that space and time are related, STARnet jointly optimizes three tasks (i.e., space SR, time SR, and space-time SR).…”

Section: Ttimentioning

confidence: 99%