Recurrent Back-Projection Network for Video Super-Resolution

Haris, Muhammad; Shakhnarovich, Gregory; Ukita, Norimichi

doi:10.1109/cvpr.2019.00402

Cited by 483 publications

(384 citation statements)

References 37 publications

(112 reference statements)

Supporting

Mentioning

384

Contrasting

Order By: Relevance

“…We compare our EDVR method with nine algorithms: RCAN [52], DeepSR [19], BayesSR [21], VESPCN [2], SPMC [37], TOFlow [48], FRVSR [32], DUF [10] and RBPN [6] on three testing datasets: Vid4 [21], Vimeo-90K-T [48] and REDS4. Most previous methods use different training sets and different down-sampling kernels, making the comparisons difficult.…”

Section: Comparisons With State-of-the-art Methodsmentioning

confidence: 99%

EDVR: Video Restoration With Enhanced Deformable Convolutional Networks

Wang

Chan

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

1,051

1,017

View full text Add to dashboard Cite

Video restoration tasks, including super-resolution, deblurring, etc, are drawing increasing attention in the computer vision community. A challenging benchmark named REDS is released in the NTIRE19 Challenge. This new benchmark challenges existing methods from two aspects:(1) how to align multiple frames given large motions, and (2) how to effectively fuse different frames with diverse motion and blur. In this work, we propose a novel Video Restoration framework with Enhanced Deformable convolutions, termed EDVR, to address these challenges. First, to handle large motions, we devise a Pyramid, Cascading and Deformable (PCD) alignment module, in which frame alignment is done at the feature level using deformable convolutions in a coarse-to-fine manner. Second, we propose a Temporal and Spatial Attention (TSA) fusion module, in which attention is applied both temporally and spatially, so as to emphasize important features for subsequent restoration. Thanks to these modules, our EDVR wins the champions and outperforms the second place by a large margin in all four tracks in the NTIRE19 video restoration and enhancement challenges. EDVR also demonstrates superior performance to state-of-the-art published methods on video super-resolution and deblurring. The code is available at https://github.com/xinntao/EDVR.

show abstract

Section: Comparisons With State-of-the-art Methodsmentioning

confidence: 99%

EDVR: Video Restoration With Enhanced Deformable Convolutional Networks

Wang

Chan

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

1,051

1,017

View full text Add to dashboard Cite

show abstract

“…We compare our DNLN with several state-of-theart SISR and VSR methods: DBPN [5], RCAN [27], VESPCN [1], TOFlow [26], FRVSR [18], DUF [8] and RBPN [6]. Note that most previous methods are trained with different datasets and we just compare with the results they provided.…”

Section: Comparison With the State-of-the-art Methodsmentioning

confidence: 99%

“…Although DUF and RBPN could reproduce part of the HR patterns compared with other methods, it is obvious that our DNLN is the unique approach to restore the abundant details and clean edges. Table 3 presents the quantitative outcomes of Vimeo-90K-T. As suggested in [6], we classified the video clips into three different groups (e.g. slow, medium and fast) according to the motion velocity.…”

Section: Comparison With the State-of-the-art Methodsmentioning

confidence: 99%

Deformable Non-Local Network for Video Super-Resolution

Wang

Liu

et al. 2019

IEEE Access

View full text Add to dashboard Cite

The video super-resolution (VSR) task aims to restore a high-resolution video frame by using its corresponding low-resolution frame and multiple neighboring frames. At present, many deep learning-based VSR methods rely on optical flow to perform frame alignment. The final recovery results will be greatly affected by the accuracy of optical flow. However, optical flow estimation cannot be completely accurate, and there are always some errors. In this paper, we propose a novel deformable non-local network (DNLN) which is non-flow-based. Specifically, we apply the improved deformable convolution in our alignment module to achieve adaptive frame alignment at the feature level. Furthermore, we utilize a non-local module to capture the global correlation between the reference frame and aligned neighboring frame, and simultaneously enhance desired fine details in the aligned frame. To reconstruct the final high-quality HR video frames, we use residual in residual dense blocks to take full advantage of the hierarchical features. Experimental results on several datasets demonstrate that the proposed DNLN can achieve state of the art performance on video super-resolution task.

show abstract

“…Due to this mixture, the CNN may generate the blurry HR frame. Recently techniques [20][21][22] presented pipeline architecture to address these challenges. The pipeline architecture follows feature extraction, feature alignment, fusion and frame reconstruction.…”

Section: Align-filter and Learn Video Super Resolution Using Deep Learnmentioning

confidence: 99%

“…However, this method uses a single frame processing which leads to generate the suboptimal outcome in case of occlusion. Recently, Haris et al [22] addressed this issues and developed recurrent back-projection network (RBPN) for SR. In this method, the network processes multiple frames where spatial and temporal information is extracted and fused using recurrent encoder-decoder module.…”

Section: Literature Surveymentioning

confidence: 99%

Align-Filter & Learn Video Super Resolution using Deep learning (AFLVSR)

2019

IJRTE

View full text Add to dashboard Cite

In this article, we focused on the super-resolution technique in computer vision applications. During last decade, image super-resolution techniques have been introduced and adopted widely in various applications. However, increasing demand of high quality multimedia data has lead towards the high resolution data streaming. The conventional techniques which are based on the image super-resolution are not suitable for multi-frame SR. Moreover, the motion estimation, motion compensation, spatial and temporal information extraction are the well-known challenging issues in video super-resolution field. In this work, we address these issues and developed deep learning based novel architecture which performs feature alignment, filtering the image using deep learning and estimates the residual of low-resolution frames to generate the high-resolution frame. The proposed approach is named as Align-Filter & Learn Video Super resolution using Deep learning (AFLVSR). We have conducted and extensive experimental analysis which shows a significant improvement in the performance when compared with the state-of-art video SR techniques.

show abstract

Recurrent Back-Projection Network for Video Super-Resolution

Cited by 483 publications

References 37 publications

EDVR: Video Restoration With Enhanced Deformable Convolutional Networks

EDVR: Video Restoration With Enhanced Deformable Convolutional Networks

Deformable Non-Local Network for Video Super-Resolution

Align-Filter & Learn Video Super Resolution using Deep learning (AFLVSR)

Contact Info

Product

Resources

About