FAST: A Framework to Accelerate Super-Resolution Processing on Compressed Videos

Zhang, Zhengdong; Sze, Vivienne

doi:10.1109/cvprw.2017.138

Cited by 37 publications

(14 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Future work can integrate camera sensors that avoid spending energy to capture redundant data [28,[62][63][64], and end-to-end visual applications can inform the system about which semantic changes are relevant for their task. A change-oriented visual system could exploit the motion vectors that hardware video codecs already produce, as recent work has done for super-resolution [26]. Through holistic co-design, approximately incremental vision can enable systems that spend resources in proportion to relevant events in the environment.…”

Section: Discussionmentioning

confidence: 99%

EVA²: Exploiting Temporal Redundancy in Live Computer Vision

Buckler

Bedoukian

Jayasuriya

et al. 2018

2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA)

View full text Add to dashboard Cite

Hardware support for deep convolutional neural networks (CNNs) is critical to advanced computer vision in mobile and embedded devices. Current designs, however, accelerate generic CNNs; they do not exploit the unique characteristics of real-time vision. We propose to use the temporal redundancy in natural video to avoid unnecessary computation on most frames. A new algorithm, activation motion compensation, detects changes in the visual input and incrementally updates a previously-computed activation. The technique takes inspiration from video compression and applies well-known motion estimation techniques to adapt to visual changes. We use an adaptive key frame rate to control the trade-off between efficiency and vision quality as the input changes. We implement the technique in hardware as an extension to state-of-the-art CNN accelerator designs. The new unit reduces the average energy per frame by 54%, 62%, and 87% for three CNNs with less than 1% loss in vision accuracy.

show abstract

Section: Discussionmentioning

confidence: 99%

EVA²: Exploiting Temporal Redundancy in Live Computer Vision

Buckler

Bedoukian

Jayasuriya

et al. 2018

2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA)

View full text Add to dashboard Cite

show abstract

“…Although these methods improved visual quality, these methods are slower than many CNN-based VSR methods. To overcome this issue, Zhang et al [33] used pixel correlations extracted by compression algorithms to exploit dense representation of the network; by transferring the SR result between adjacent frames, they accelerated the VSR process by almost 15 times with little performance loss. Another method proposed by Xue et al [6] was a turning point for the optical flow-based method for VSR.…”

Section: Video Super-resolution (Vsr)mentioning

confidence: 99%

DSTnet: Deformable Spatio-Temporal Convolutional Residual Network for Video Super-Resolution

2021

View full text Add to dashboard Cite

Video super-resolution (VSR) aims at generating high-resolution (HR) video frames with plausible and temporally consistent details using their low-resolution (LR) counterparts, and neighboring frames. The key challenge for VSR lies in the effective exploitation of intra-frame spatial relation and temporal dependency between consecutive frames. Many existing techniques utilize spatial and temporal information separately and compensate motion via alignment. These methods cannot fully exploit the spatio-temporal information that significantly affects the quality of resultant HR videos. In this work, a novel deformable spatio-temporal convolutional residual network (DSTnet) is proposed to overcome the issues of separate motion estimation and compensation methods for VSR. The proposed framework consists of 3D convolutional residual blocks decomposed into spatial and temporal (2+1) D streams. This decomposition can simultaneously utilize input video’s spatial and temporal features without a separate motion estimation and compensation module. Furthermore, the deformable convolution layers have been used in the proposed model that enhances its motion-awareness capability. Our contribution is twofold; firstly, the proposed approach can overcome the challenges in modeling complex motions by efficiently using spatio-temporal information. Secondly, the proposed model has fewer parameters to learn than state-of-the-art methods, making it a computationally lean and efficient framework for VSR. Experiments are conducted on a benchmark Vid4 dataset to evaluate the efficacy of the proposed approach. The results demonstrate that the proposed approach achieves superior quantitative and qualitative performance compared to the state-of-the-art methods.

show abstract

“…The field of image SR is especially attractive. The image super-resolution has many applications, such as medical image processing [7], [8], facial image improvement [9], [10], enhancement of compressed images/videos [11], [12], and thermal image enhancement [13]. Nevertheless, the SR is a broader concept.…”

Section: Introductionmentioning

confidence: 99%

Temporal Graph Super Resolution on Power Distribution Network Measurements

et al. 2021

View full text Add to dashboard Cite

The applications of super-resolution (SR) technology in the field of image completion are successful. Nevertheless, industry applications demand not only image completion but also the topology and time-series completion. In this paper, the SR technology on a topology graph is studied in the scenario of recovering measurements in power distribution systems for cost saving and security & stability improvement. The power flow and voltage magnitude measurements on feeders are reported at different frequencies. In this paper, a new data completion method considering distribution system topology is proposed. Firstly, the graph convolutional neural network (GCN) is used for spatial-temporal convolution on a graph, and then the power system state estimation (SE) is used introducing the physical constraints. This method realizes the super-resolution of distribution system measurements, improves the state awareness of distribution systems. Hence, it helps to improve the efficiency of distribution network operation and to reduce equipment failures.

show abstract

FAST: A Framework to Accelerate Super-Resolution Processing on Compressed Videos

Cited by 37 publications

References 35 publications

EVA²: Exploiting Temporal Redundancy in Live Computer Vision

EVA²: Exploiting Temporal Redundancy in Live Computer Vision

DSTnet: Deformable Spatio-Temporal Convolutional Residual Network for Video Super-Resolution

Temporal Graph Super Resolution on Power Distribution Network Measurements

Contact Info

Product

Resources

About