Deep Video Deblurring: The Devil is in the Details

Gast, Jochen; Roth, Stefan

doi:10.1109/iccvw.2019.00475

Cited by 23 publications

(9 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Optical flow estimators are widely used in video restoration tasks (Gast & Roth, 2019;Xue et al, 2019;Gong et al, 2017;Sun et al, 2015;Makansi et al, 2017;Su et al, 2017;Pan et al, 2020) to align highly related but mis-aligned frames. Previous flow-based video deblurring methods (Xue et al, 2019;Makansi et al, 2017;Su et al, 2017;Pan et al, 2020;Gast & Roth, 2019) mainly adopt the pre-warping strategy, which firstly estimates the optical flow and then warps the neighboring frames. For example, (Su et al, 2017) experiments with pre-warping input images based on classic optical flow methods to register them to the reference frame.…”

Section: Flow-based Video Restorationmentioning

confidence: 99%

Flow-Guided Sparse Transformer for Video Deblurring

Jiang¹,

Cai²,

Hu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Exploiting similar and sharper scene patches in spatio-temporal neighborhoods is critical for video deblurring. However, CNN-based methods show limitations in capturing long-range dependencies and modeling non-local self-similarity. In this paper, we propose a novel framework, Flow-Guided Sparse Transformer (FGST), for video deblurring. In FGST, we customize a selfattention module, Flow-Guided Sparse Windowbased Multi-head Self-Attention (FGSW-MSA). For each query element on the blurry reference frame, FGSW-MSA enjoys the guidance of the estimated optical flow to globally sample spatially sparse yet highly related key elements corresponding to the same scene patch in neighboring frames. Besides, we present a Recurrent Embedding (RE) mechanism to transfer information from past frames and strengthen long-range temporal dependencies. Comprehensive experiments demonstrate that our proposed FGST outperforms state-of-the-art (SOTA) methods on both DVD and GOPRO datasets and even yields more visually pleasing results in real video deblurring. Code and models will be released to the public.

show abstract

Section: Flow-based Video Restorationmentioning

confidence: 99%

Flow-Guided Sparse Transformer for Video Deblurring

Jiang¹,

Cai²,

Hu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The learning of the optimal burst exposure parameters ∆ mandates a differentiable calculation of the forward model described by equations (5)(6)(7)(8)(9)(10)(11)(12). While most of these computations are straightforward, special consideration should be taken in several steps detailed below.…”

Section: Numerical Approximationmentioning

confidence: 99%

“…Since fixed-size kernels are limited in their ability to align frames, we pre-warp the burst according to a reference one, as customarily applied in video deblurring works [18,37]. Gast and Roth [8] proposed using pretrained flow networks as the most efficient option for the latter operation. However, this approach is impractical in our case, as the amount of noise and blur in the input frames is expected to change while the exposure times are learned.…”

Section: Reconstruction Networkmentioning

confidence: 99%

Digital Gimbal: End-to-end Deep Image Stabilization with Learnable Exposure Times

Dahary¹,

Jacoby²,

Bronstein³

2020

Preprint

View full text Add to dashboard Cite

Mechanical image stabilization using actuated gimbals enables capturing long-exposure shots without suffering from blur due to camera motion. These devices, however, are often physically cumbersome and expensive, limiting their widespread use. In this work, we propose to digitally emulate a mechanically stabilized system from the input of a fast unstabilized camera. To exploit the trade-off between motion blur at long exposures and low SNR at short exposures, we train a CNN that estimates a sharp high-SNR image by aggregating a burst of noisy short-exposure frames, related by unknown motion. We further suggest learning the burst's exposure times in an end-to-end manner, thus balancing the noise and blur across the frames. We demonstrate this method's advantage over the traditional approach of deblurring a single image or denoising a fixedexposure burst.

show abstract

“…The first family aims at focusing on one single subtask of video enhancement. For example, some methods [49,5,55] are suggested for video compression artifacts removal, some [28,48,31,13,33] are designed for video denoising, some [2,52,56,21,32,38,20,37] are proposed for frame interpolation, and some [44,1,11] are developed for video deblurring. Besides, some [29,22,30,50,46,41,6,4] aim to enhance the resolution of a given video by adding missing high-frequency information.…”

Section: Related Workmentioning

confidence: 99%

An Efficient Recurrent Adversarial Framework for Unsupervised Real-Time Video Enhancement

Fuoli¹,

Huang²,

Paudel³

et al. 2020

Preprint

View full text Add to dashboard Cite

Video enhancement is a challenging problem, more than that of stills, mainly due to high computational cost, larger data volumes and the difficulty of achieving consistency in the spatio-temporal domain. In practice, these challenges are often coupled with the lack of example pairs, which inhibits the application of supervised learning strategies. To address these challenges, we propose an efficient adversarial video enhancement framework that learns directly from unpaired video examples. In particular, our framework introduces new recurrent cells that consist of interleaved local and global modules for implicit integration of spatial and temporal information. The proposed design allows our recurrent cells to efficiently propagate spatio-temporal information across frames and reduces the need for high complexity networks. Our setting enables learning from unpaired videos in a cyclic adversarial manner, where the proposed recurrent units are employed in all architectures. Efficient training is accomplished by introducing one single discriminator that learns the joint distribution of source and target domain simultaneously. The enhancement results demonstrate clear superiority of the proposed video enhancer over the state-of-theart methods, in all terms of visual quality, quantitative metrics, and inference speed. Notably, our video enhancer is capable of enhancing over 35 frames per second of FullHD video (1080x1920).

show abstract

Deep Video Deblurring: The Devil is in the Details

Cited by 23 publications

References 54 publications

Flow-Guided Sparse Transformer for Video Deblurring

Flow-Guided Sparse Transformer for Video Deblurring

Digital Gimbal: End-to-end Deep Image Stabilization with Learnable Exposure Times

An Efficient Recurrent Adversarial Framework for Unsupervised Real-Time Video Enhancement

Contact Info

Product

Resources

About