Dictionary-based multiple frame video super-resolution

Dai, Qiqin; Yoo, Seunghwan; Kappeler, Armin; Katsaggelos, Aggelos K.

doi:10.1109/icip.2015.7350764

Cited by 24 publications

(15 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Kernel regression methods [35] have been shown to be applicable to videos using 3D kernels instead of 2D ones [36]. Dictionary learning approaches, which define LR images as a sparse linear combination of dictionary atoms coupled to a HR dictionary, have also been adapted from images [38] to videos [4]. Another approach is example-based patch recurrence, which assumes patches in a single image or video obey multi-scale relationships, and therefore missing high-frequency content at a given scale can be inferred from coarser scale patches.…”

Section: Related Workmentioning

confidence: 99%

Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation

Caballero

Ledig

Aitken

et al. 2017

2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

661

553

View full text Add to dashboard Cite

Convolutional neural networks have enabled accurate image super-resolution in real-time. However, recent attempts to benefit from temporal correlations in video superresolution have been limited to naive or inefficient architectures. In this paper, we introduce spatio-temporal subpixel convolution networks that effectively exploit temporal redundancies and improve reconstruction accuracy while maintaining real-time speed. Specifically, we discuss the use of early fusion, slow fusion and 3D convolutions for the joint processing of multiple consecutive video frames. We also propose a novel joint motion compensation and video super-resolution algorithm that is orders of magnitude more efficient than competing methods, relying on a fast multi-resolution spatial transformer module that is endto-end trainable. These contributions provide both higher accuracy and temporally more consistent videos, which we confirm qualitatively and quantitatively. Relative to singleframe models, spatio-temporal networks can either reduce the computational cost by 30% whilst maintaining the same quality or provide a 0.2dB gain for a similar computational cost. Results on publicly available datasets demonstrate that the proposed algorithms surpass current state-of-theart performance in both accuracy and efficiency.

show abstract

Section: Related Workmentioning

confidence: 99%

Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation

Caballero

Ledig

Aitken

et al. 2017

2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

661

553

View full text Add to dashboard Cite

show abstract

“…The successes encourage the community to further attempt deep learning on the more challenging video restoration problems. Earlier studies [36,4,33,19,11] treat video restoration as a simple exten-Bicubic RCAN…”

Section: Introductionmentioning

confidence: 99%

EDVR: Video Restoration With Enhanced Deformable Convolutional Networks

Wang

Chan

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

1,051

1,017

View full text Add to dashboard Cite

Video restoration tasks, including super-resolution, deblurring, etc, are drawing increasing attention in the computer vision community. A challenging benchmark named REDS is released in the NTIRE19 Challenge. This new benchmark challenges existing methods from two aspects:(1) how to align multiple frames given large motions, and (2) how to effectively fuse different frames with diverse motion and blur. In this work, we propose a novel Video Restoration framework with Enhanced Deformable convolutions, termed EDVR, to address these challenges. First, to handle large motions, we devise a Pyramid, Cascading and Deformable (PCD) alignment module, in which frame alignment is done at the feature level using deformable convolutions in a coarse-to-fine manner. Second, we propose a Temporal and Spatial Attention (TSA) fusion module, in which attention is applied both temporally and spatially, so as to emphasize important features for subsequent restoration. Thanks to these modules, our EDVR wins the champions and outperforms the second place by a large margin in all four tracks in the NTIRE19 video restoration and enhancement challenges. EDVR also demonstrates superior performance to state-of-the-art published methods on video super-resolution and deblurring. The code is available at https://github.com/xinntao/EDVR.

show abstract

“…Kato et al [12] and Dai et al [4] presented approaches to utilize multiple low resolution frames for generating one target high resolution frame. In multi-frame SR, a set of low resolution frames Y i , i = 1, 2, .…”

Section: B Multi-frame Srmentioning

confidence: 99%

“…Recently, example based techniques were improved further by introducing sparse coding [15] to the SR problem. Yang et al [23], [24], [25] improved the approach presented in [3] by employing sparse representation paradigm and Dai et al [4] and Kato et al [12] made further contributions in the enhancement of the sparse representation based SR.…”

Section: Introductionmentioning

confidence: 99%

An efficient video super-resolution approach based on sparse representation

Toutounchi¹,

Ones²,

Izquierdo³

2017

2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP)

View full text Add to dashboard Cite

Block-wise super-resolution methods, and in particular sparse representation based approaches, often focus on spatial upsampling of still images. Applying such models to videos is an extremely time consuming process due to the expensive sparse coding process for every block of each frame, and the conventional exhaustive overlapping blocks processing for reducing the blocking artifacts. In this paper, we introduce an approach that enables us to skip the sparse representation process for the static portions of a video by predicting the high resolution blocks from previously super-resolved frames, exploiting the significant amount of correlation between the adjacent frames in a video. Our approach is also enhanced by adaptive in-loop filters for removing the blocking and pixel-wise artifacts, replacing the overlapping blocks structure of superresolution. Our method provides comparable results in terms of image quality with respect to the state of the art, and can reduce the processing time of the super-resolution methods by 63% in average. It enables the block based super-resolution methods to be applied in upsampling of videos to very high resolutions with a reasonable processing time.

show abstract

Dictionary-based multiple frame video super-resolution

Cited by 24 publications

References 17 publications

Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation

Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation

EDVR: Video Restoration With Enhanced Deformable Convolutional Networks

An efficient video super-resolution approach based on sparse representation

Contact Info

Product

Resources

About