Recurrent Video Deblurring with Blur-Invariant Motion Estimation and Pixel Volumes

Abstract

For the success of video deblurring, it is essential to utilize information from neighboring frames. Most state-of-the-art video deblurring methods adopt motion compensation between video frames to aggregate information from multiple frames that can help deblur a target frame. However, the motion compensation methods adopted by previous deblurring methods are not blur-invariant, and consequently, their accuracy is limited for blurry frames with different blur amounts. To alleviate this problem, we propose two novel approaches to deblur videos by effectively aggregating information from multiple video frames. First, we present blur-invariant motion estimation learning to improve motion estimation accuracy between blurry frames. Second, for motion compensation, instead of aligning frames by warping with estimated motions, we use a pixel volume that contains candidate sharp pixels to resolve motion estimation errors. We combine these two processes to propose an effective recurrent video deblurring network that fully exploits deblurred previous frames. Experiments show that our method achieves the state-of-the-art performance both quantitatively and qualitatively compared to recent methods that use deep learning.

VDTR: Video Deblurring with Transformer

“…These RNN-based models adopt fewer frames for temporal modeling, making the sharp information limited. VDTR also obtains 1.2dB higher PSNR compared to EDVR, which utilizes deformable convolution to align [11] and the state-of-the-art CNN-based video deblurring methods CDVDTSP [12] and PVDNet [44] (these two methods achieved highest PSNRs and SSIMs) on DVD [11] datasets. VDTR demonstrates strong competitiveness.…”

“…All experiments are conducted on 8 NVIDIA Tesla V100 GPU with 32G memory. c) Evaluation Metrics: We compared VDTR with stateof-the-art convolution-based networks, including single image deblurring methods SRN [5], video deblurring methods DBN [11], DBLRNet [30], IFIRNN [29], EDVR [13], STFAN [14], ESTRNN [22], CDVDTSP [12], PVDNet [44] quantitatively and qualitatively. We adopt public available source codes for evaluation.…”

Pyramid Feature Alignment Network for Video Deblurring

“…In this section, we take the model trained on the GoPro [28] using three blurry frames for reconstruction to analyze the effects of multiple components in our framework. [46] (c) TSP [30] (d) PVDNet [35] (e) PFAN (Ours) (f) Ground Truth…”

“…To evaluate the performance of the proposed method, we compare it against state-of-the-art methods [9,24,28,30,[35][36][37]40,41,46]. To evaluate the quality of each restored image on the datasets, we use PSNR and SSIM as the evaluation metrics.…”

“…Li et al [24] proposed to adopt a pyramid correlation volume to learn the spatial correspondence among blurry frames. Son et al [35] presented blur-invariant motion estimation methods and a way to resolve motion estimation errors. However, these methods for information aggregation focused at a particular scale while ignoring to explore multi-scale information for better alignment and reconstruction.…”