Recurrent Neural Networks With Intra-Frame Iterations for Video Deblurring

Nah, Seungjun; Son, Sanghyun; Lee, Kyoung Mu

doi:10.1109/cvpr.2019.00829

Cited by 140 publications

(124 citation statements)

References 29 publications

Supporting

Mentioning

123

Contrasting

Order By: Relevance

“…The CNN-based methods [4], [6] make an inference of the deblurred frame by stacking neighboring frames with current frame as input to the CNN framework. The RNNbased methods, like [3], [5], [7], [8], employ recurrent neural network architecture to transfer the effective information frame by frame for deblurring. However, how to utilize spatio-temporal dependency of video for deblurring more efficiently still needs to be explored.…”

Section: Introductionmentioning

confidence: 99%

“…A comparison of network efficiency on video deblurring. SRN [1], DeepDeblur [2] are methods for image deblurring, and STRCNN [3], DBN [4], IFIRNN [5] are methods for video deblurring. (a) shows the computational cost required for processing a frame of 720P(1280 × 720) video and the corresponding performance of each model on GOPRO [2] dataset in terms of GMACs and PSNR, respectively.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Efficient Spatio-Temporal Recurrent Neural Network for Video Deblurring

Zhong

Gao²,

Zheng

et al. 2020

Lecture Notes in Computer Science

113

126

View full text Add to dashboard Cite

Real-time video deblurring still remains a challenging task due to the complexity of spatially and temporally varying blur itself and the requirement of low computational cost. To improve the network efficiency, we adopt residual dense blocks into RNN cells, so as to efficiently extract the spatial features of the current frame. Furthermore, a global spatio-temporal attention module is proposed to fuse the effective hierarchical features from past and future frames to help better deblur the current frame. Another issue needs to be addressed urgently is the lack of a real-world benchmark dataset. Thus, we contribute a novel dataset (BSD) to the community, by collecting paired blurry/sharp video clips using a co-axis beam splitter acquisition system. Experimental results show that the proposed method (ESTRNN) can achieve better deblurring performance both quantitatively and qualitatively with less computational cost against state-of-the-art video deblurring methods. In addition, cross-validation experiments between datasets illustrate the high generality of BSD over the synthetic datasets. The code and dataset are released at https://github.com/zzh-tech/ESTRNN.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Efficient Spatio-Temporal Recurrent Neural Network for Video Deblurring

Zhong

Gao²,

Zheng

et al. 2020

Lecture Notes in Computer Science

113

126

View full text Add to dashboard Cite

show abstract

“…In order to demonstrate the effectiveness of our proposed network, we compare it to some state-of-the-art methods such as PSDEBLUR, DeblurGAN [48], MSCNN [6], WFA [49], DBN [4], STAN(M/A_A) [50], DMPHN [51], RNNs [45] and IFI-RNN [52]. In Table 4, PSDEBLUR is the deblurred results of PHOTOSHOP, and INPUT represents the blurry images.…”

Section: Comparisonmentioning

confidence: 99%

“…STAN [50] uses a motion estimation and motion compensation module to warp the previous deblurred frame to restore the current frame. The method IFI-RNN [52] is also a recurrent neural network aims at video deblurring. However, the most difference between the IFI-RNN and our method is that hidden states of our model is provided by the two weight generators rather than transformation from former recurrent cells.…”

Section: Comparisonmentioning

confidence: 99%

Video Deblurring via Temporally and Spatially Variant Recurrent Neural Network

et al. 2020

View full text Add to dashboard Cite

The camera shake and high-speed motion of objects often produce a blurry video. However, it is hard to recover sharp videos using existing single or multiple image deblurring methods, as the blur artifacts in blurry videos are both temporally and spatially varying. In this paper, we propose a temporally and spatially variant recurrent neural network for video deblurring, in which both temporally and spatially variants employ ConvGRU blocks and a weight generator to capture spatio-temporal features. Meanwhile, the proposed model is trained in an end-to-end manner, where the model input and output are set to the same number. Thus, our model does not reduce the number of frames both in training and testing stages, which is important in practical applications. Quantitative and qualitative evaluations on standard benchmark datasets demonstrate that the proposed method outperforms the current state-of-the-art methods. INDEX TERMS Spatio-temporal features, video deblurring, variant recurrent neural network.

show abstract

“…Kim et al [21] focus on the temporal nature of the problem by applying a temporal feature blending layer within an RNN. Similarly, Nah et al [35] apply an RNN to propagate intra-frame information. While RNNs are promising, we note that these are often difficult to train in practice [38].…”

Section: Related Workmentioning

confidence: 99%

Deep Video Deblurring: The Devil is in the Details

Gast

Roth

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

Video deblurring for hand-held cameras is a challenging task, since the underlying blur is caused by both camera shake and object motion. State-of-the-art deep networks exploit temporal information from neighboring frames, either by means of spatio-temporal transformers or by recurrent architectures. In contrast to these involved models, we found that a simple baseline CNN can perform astonishingly well when particular care is taken w.r.t. the details of model and training procedure. To that end, we conduct a comprehensive study regarding these crucial details, uncovering extreme differences in quantitative and qualitative performance. Exploiting these details allows us to boost the architecture and training procedure of a simple baseline CNN by a staggering 3.15dB, such that it becomes highly competitive w.r.t. cutting-edge networks. This raises the question whether the reported accuracy difference between models is always due to technical contributions or also subject to such orthogonal, but crucial details.

show abstract

Recurrent Neural Networks With Intra-Frame Iterations for Video Deblurring

Cited by 140 publications

References 29 publications

Efficient Spatio-Temporal Recurrent Neural Network for Video Deblurring

Efficient Spatio-Temporal Recurrent Neural Network for Video Deblurring

Video Deblurring via Temporally and Spatially Variant Recurrent Neural Network

Deep Video Deblurring: The Devil is in the Details

Contact Info

Product

Resources

About