Generic 3D Convolutional Fusion for Image Restoration

Abstract

Also recently, exciting strides forward have been made in the area of image restoration, particularly for image denoising and single image super-resolution. Deep learning techniques contributed to this significantly. The top methods differ in their formulations and assumptions, so even if their average performance may be similar, some work better on certain image types and image regions than others. This complementarity motivated us to propose a novel 3D convolutional fusion (3DCF) method. Unlike other methods adapted to different tasks, our method uses the exact same convolutional network architecture to address both image denoising and single image super-resolution. As a result, our 3DCF method achieves substantial improvements (0.1dB-0.4dB PSNR) over the state-of-the-art methods that it fuses, and this on standard benchmarks for both tasks. At the same time, the method still is computationally efficient.

CocoNet: A Deep Neural Network for Mapping Pixel Coordinates to Color Values

“…To the best of our knowledge, there are no previous scientific works that propose to learn a mapping of the pixel coordinates to the corresponding pixel color values using neural networks. However, there are numerous neural models that learn a mapping from image pixels to a set of classes [10,16,30,33] or from pixels to pixels [1,2,[4][5][6]12,14,18,20,21,24,28,29,35,[39][40][41][42]. The neural models that map pixels to pixels are usually applied on tasks such as image compression [1,2,4,6,21,24,35], image denoising and restoration [20,39,42], image super-resolution [5,14,18,20,28,29,39,40], image completion [12,41] and image generation [11,36].…”

“…There are models that address multiple aspects of image restoration [20,39,42], but there are some models that deal with specific sub-tasks, e.g. image super-resolution [5,14,18,20,28,29,34,39,40]. Zhao et al [42] propose a deep cascade of neural networks to solve the inpainting, deblurring, denoising problems at the same time.…”

“…Their model contains two networks, an inpainting generative adversarial network (GAN) and a deblurring-denoising network. Wu et al [39] propose a novel 3D convolutional fusion method to address both image denoising and single image super-resolution. To address the same tasks, Mao et al [20] describe a fully convolutional encoding-decoding framework composed of multiple layers of convolution and deconvolution operators.…”

“…However, there are numerous neural models that learn a mapping from image pixels to a set of classes [10,16,30,33] or from pixels to pixels [1,2,[4][5][6]12,14,18,20,21,24,28,29,35,[39][40][41][42]. The neural models that map pixels to pixels are usually applied on tasks such as image compression [1,2,4,6,21,24,35], image denoising and restoration [20,39,42], image super-resolution [5,14,18,20,28,29,39,40], image completion [12,41] and image generation [11,36]. Since our approach, CocoNet, can be applied to similar tasks, we consider that the neural models that map pixels to pixels are more closely related to ours.…”

“…Hence, image denoising and image super-resolution can be regarded as sub-tasks of image restoration. There are models that address multiple aspects of image restoration [20,39,42], but there are some models that deal with specific sub-tasks, e.g. image super-resolution [5,14,18,20,28,29,34,39,40].…”