Iterative Residual CNNs for Burst Photography Applications

Abstract

Modern inexpensive imaging sensors suffer from inherent hardware constraints which often result in captured images of poor quality. Among the most common ways to deal with such limitations is to rely on burst photography, which nowadays acts as the backbone of all modern smartphone imaging applications. In this work, we focus on the fact that every frame of a burst sequence can be accurately described by a forward (physical) model. This, in turn, allows us to restore a single image of higher quality from a sequence of low-quality images as the solution of an optimization problem. Inspired by an extension of the gradient descent method that can handle non-smooth functions, namely the proximal gradient descent, and modern deep learning techniques, we propose a convolutional iterative network with a transparent architecture. Our network uses a burst of lowquality image frames and is able to produce an output of higher image quality recovering fine details which are not distinguishable in any of the original burst frames. We focus both on the burst photography pipeline as a whole, i.e., burst demosaicking and denoising, as well as on the traditional Gaussian denoising task. The developed method demonstrates consistent state-of-the art performance across the two tasks and as opposed to other recent deep learning approaches does not have any inherent restrictions either to the number of frames or their ordering.

Deep Reparametrization of Multi-Frame Super-Resolution and Denoising

“…Deep Optimization-based image restoration: A number of deep learning based approaches [35,36,65,66] have posed image restoration tasks as an explicit optimization problem. The P 3 [61] and RED [50] approaches provide a general framework for utilizing standard denoising methods as regularizers in optimization-based image restoration methods.…”

“…Zhang et al [66] used the half quadratic splitting method to plug a deep neural-network based denoiser prior into model-based optimization methods. Kokkinos et al [36] used a proximal gradient descent based framework to learn a regularizer network for burst photography applications. These prior works mainly focus on only learning the regularizer, while assuming that the data term (image formation process) is known and simple.…”

“…As a widely embraced paradigm [15,17,36,47], MFIR is addressed by first modelling the image formation process as, x i = H φ mi (y) + η i . In this model, the original image y is affected by the scene motion φ mi , image degradation H, and noise η i , resulting in the observed image x i .…”

RBSRICNN: Raw Burst Super-Resolution through Iterative Convolutional Neural Network

“…Due to the ill-posed nature of the SISR problem, the existing methods have limited performance to recover high frequency details through single image learned priors. Besides the development of the SISR approaches based on single image priors, the recent works [11,17,18,33,6,3] have demonstrated the potential of MFSR methods that aim to fuse multiple LR frames to reconstruct a HR output. However, the deep learning based Raw Burst SR methods are black-box data-driven approaches with larger model size, while our proposed scheme (RBSRICNN) has the merit of interpretability and small model size with good reconstruction results as shown in 2 Proposed Methodology…”

Multi-Kernel Prediction Networks for Denoising of Burst Images

“…Burst denoising methods commonly first align the successive images as a pre-processing step, and then fuse and denoise the aligned images [15,2,1,16]. Many of the state-of-the-art approaches in burst image denoising are based on fully convolutional neural network architectures [1,14,16,17].…”