Real-Time Video Super-Resolution with Spatio-Temporal Modeling and Redundancy-Aware Inference

Abstract

Video super-resolution aims to generate high-resolution frames from low-resolution counterparts. It can be regarded as a specialized application of image super-resolution, serving various purposes, such as video display and surveillance. This paper proposes a novel method for real-time video super-resolution. It effectively exploits spatial information by utilizing the capabilities of an image super-resolution model and leverages the temporal information inherent in videos. Specifically, the method incorporates a pre-trained image super-resolution network as its foundational framework, allowing it to leverage existing expertise for super-resolution. A fast temporal information aggregation module is presented to further aggregate temporal cues across frames. By using deformable convolution to align features of neighboring frames, this module takes advantage of inter-frame dependency. In addition, it employs a hierarchical fast spatial offset feature extraction and a channel attention-based temporal fusion. A redundancy-aware inference algorithm is developed to reduce computational redundancy by reusing intermediate features, achieving real-time inferring speed. Extensive experiments on several benchmarks demonstrate that the proposed method can reconstruct satisfactory results with strong quantitative performance and visual qualities. The real-time inferring ability makes it suitable for real-world deployment.

A Survey on Video Streaming for Next-Generation Vehicular Networks

“…Wen et al [160] introduced an end-to-end deep convolutional network that dynamically generates spatial adaptive filters for aligning frames and utilizes the aligned frames to recover high-resolution frames. Wang et al [161] presented a novel low-complexity VSR method that is designed for live video applications. By efficiently leveraging spatial information from the image SR model and the inherent temporal information in the video, their method minimizes computational redundancy through the recycling of intermediate features, achieving real-time inference speed.…”

“…All videos will incorporate VSR technology [161]. Initially, low-resolution versions of the videos and VSR models will be transmitted to the user's end.…”

A Lightweight Recurrent Grouping Attention Network for Video Super-Resolution

“…Researchers have constructed different VSR models based on deep learning that can reconstruct high-quality videos. For example, researchers [18][19][20][21] have utilized explicit or implicit alignments to explore temporal flow between frames. This type of methodology can effectively align adjacent frames to the reference frame to extract high-quality temporal information.…”