FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation

Abstract

In this paper, we propose a state-of-the-art video denoising algorithm based on a convolutional neural network architecture. Until recently, video denoising with neural networks had been a largely under explored domain, and existing methods could not compete with the performance of the best patchbased methods. The approach we introduce in this paper, called FastDVDnet, shows similar or better performance than other state-of-the-art competitors with significantly lower computing times. In contrast to other existing neural network denoisers, our algorithm exhibits several desirable properties such as fast runtimes, and the ability to handle a wide range of noise levels with a single network model. The characteristics of its architecture make it possible to avoid using a costly motion compensation stage while achieving excellent performance. The combination between its denoising performance and lower computational load makes this algorithm attractive for practical denoising applications. We compare our method with different state-of-art algorithms, both visually and with respect to objective quality metrics.

First image then video: A two-stage network for spatiotemporal video denoising

“…Even worse, the flow calculation tends to be inaccurate when an object undergoes fast motion or when the image contrast between the foreground and background is weak, leading to the over-smoothness and loss of details along object boundaries in the denoised frames. Alternatively, ViDeNN [4] and FastDVDNet [30] propose one endto-end network to realize spatiotemporal denoising as [9] [19] [32] have done in video inpainting. Both of them feed pairs of noisy and clean videos to networks to successively reduce noise in input frames.…”

“…Unlike flow-based methods, such as PWC-Net [26] and TOFlow [36], such CNN-based spatiotemporal denoising network needs to model spatial noise and temporal deformation within one module. We realize this module via the method employed in FastDVDNet [30], which uses one block to process every three consecutive frames in the totally five frames and then feed concatenated output features of the first block to the other block as shown in Figure 3. This avoids explicit flow calculation which is sub-optimal in these flow-based models and the experiments show that the representative capability of CNNs help avoid boundary blurry appearing in TOFlow with the help of prior image denoising.…”

“…The first is trained with the Vimeo-Mixed noise for a fair comparison with TOFlow [36] and for demonstrating that our model is capable to deal with such kind of noise. The other is trained with the AWGN noise for a fair comparison with FastD-VDNet [30] and for demonstrating the superiority of our proposed prior image denoising stage.…”

“…Although only invoking image denoising algorithm for each frame is not effective, the spatiotemporal denoising like V-BM4D [17], which extends BM3D [5] by searching similar patches in both spatial and temporal dimensions, is instructive. FastDVDNet [30], one state-of-the-art video denoising algorithm, extends flow-based DVDNet [29] by employing modified U-Net [24] architectures to realize spatiotemporal denoising with the powerful representation of CNNs. This avoids explicit estimation of objection motion, but the results shown in Figure 1(d) demonstrate that the spatiotemporal denoising block still could not handle high-speed moving objects well because spatial noise distribution and temporal deformation are very different and Figure 1.…”

Supervised Raw Video Denoising with a Benchmark Dataset on Dynamic Scenes

“…The recently proposed ViDeNN [11] performs spatial denoising and temporal denoising sequentially and achieves better results than VBM4D. Tassano et al proposed DVDNet [33] and its fast version, called FastDVDnet [34] without explicit motion estimation, to deal with Gaussian noise removal with low computing complexity.…”