EDVR: Video Restoration With Enhanced Deformable Convolutional Networks

Abstract

Video restoration tasks, including super-resolution, deblurring, etc, are drawing increasing attention in the computer vision community. A challenging benchmark named REDS is released in the NTIRE19 Challenge. This new benchmark challenges existing methods from two aspects:(1) how to align multiple frames given large motions, and (2) how to effectively fuse different frames with diverse motion and blur. In this work, we propose a novel Video Restoration framework with Enhanced Deformable convolutions, termed EDVR, to address these challenges. First, to handle large motions, we devise a Pyramid, Cascading and Deformable (PCD) alignment module, in which frame alignment is done at the feature level using deformable convolutions in a coarse-to-fine manner. Second, we propose a Temporal and Spatial Attention (TSA) fusion module, in which attention is applied both temporally and spatially, so as to emphasize important features for subsequent restoration. Thanks to these modules, our EDVR wins the champions and outperforms the second place by a large margin in all four tracks in the NTIRE19 video restoration and enhancement challenges. EDVR also demonstrates superior performance to state-of-the-art published methods on video super-resolution and deblurring. The code is available at https://github.com/xinntao/EDVR.

Split-Attention Multiframe Alignment Network for Image Restoration

“…Furthermore, the dual attention net [47] combined channelwise attention and spatialwise attention to achieve enhanced performance for scene segmentation. In low-level vision, SFT-GAN [48] integrates a high-level prior into the image superresolution network using the attention mechanism, EDVR [12] utilizes attention to fuse features from aligned frames, and RCAN [49] combines channel attention and skip connections to enhance the representational ability of deep CNNs. Although attention mechanisms have proven their effectiveness in many tasks, they have not been effectively applied in the field of optical flow and image registration.…”

“…The current image registration techniques can be divided into four categories: global registration based on feature points and homography matrix [5]- [7], variational nonrigid registration [8], [9], optical-flow-based registration [10], [11] and implicit feature-domain registration [12], [13]. Among these techniques, global registration algorithms calculate a transformation matrix for the entire image.…”

Alanet

“…DeBlurNet [26] proposes to use consecutive frames stacked as input to generate a single clean central frame. ESVR [30] tries to align the features of multiple frames using a temporal and spatial fusion module for feature fusion from different layer to deblur a video. [12] proposes an integrated model to jointly predict the defocus blur, optical flow and latent frames.…”

NTIRE 2019 Challenge on Video Super-Resolution: Methods and Results

“…HelloVSR team proposes the EDVR framework [31], which takes 2N +1low-resolution frames as inputs and generates a high-resolution output, as shown in Fig. 4.…”