Deep Joint Deinterlacing and Denoising for Single Shot Dual-ISO HDR Reconstruction

Abstract

85 pages HDR (High Dynamic Range) images have traditionally been obtained by merging multiple exposures each captured with a different exposure time. However, this approach entails longer capture times and necessitates deghosting if the captured scene contains moving objects. With the advent of modern camera sensors that can perform per-pixel exposure modulation, it is now possible to capture all of the required exposures within a single shot. The new challenge then becomes how to best combine different pixels with different exposure values into a single full-resolution and low-noise HDR image. In this thesis, we propose a joint multi-exposure frame deinterlacing and denoising algorithm powered by deep convolutional neural networks (DCNN). In our algorithm, we first train two DCNNs, with one tuned for reconstructing low exposures and the other for high exposures. Each DCNN takes the same mosaicked dual-ISO input image and outputs either the low exposure or high exposure depending on the type of the network. The resulting exposures can be demosaicked and converted to the desired target color space prior to HDR assembly. Our evaluations using computational metrics as well as visual comparisons indicate that the quality of our reconstructions v significantly surpasses the state-of-the-art in this field.

Single-Shot High Dynamic Range Imaging via Multiscale Convolutional Neural Network

“…We evaluate the performance of the proposed algorithm against four conventional single-shot HDR imaging algorithms: Gu et al's algorithm [33] and Cho et al's algorithm [37] are interpolation-based algorithms, Choi et al's algorithm [39] is a sparse representation model-based algorithm [39], and An and Lee's algorithm [40] and Çogalan and Akyüz's algorithm [41] are learning-based algorithms. The performance of these algorithms is evaluated using synthetic noncalibrated and calibrated images, and captured real SVE images.…”

“…For Choi et al's algorithm [39], we used the patch size of 6 × 6 and fixed the sparsity regularization parameter λ to 0.15. For An and Lee's algorithm [40] and Çogalan and Akyüz's algorithm [41], we retrained their networks with our dataset, which will be described subsequently. For reproducibility, we provide the source codes and pretrained models on our project website.…”

“…• We experimentally show, with both synthetic and real image datasets, that the proposed CNN model trained with the robust loss function produces clear and naturallooking HDR images of significantly higher quality than the conventional algorithms [33], [37], [39]- [41].…”

“…Recently, deep learning-based approaches to single-shot HDR imaging were developed [40], [41], which can reconstruct over-and under-exposed pixels in raw SVE Bayer images. While these algorithms synthesize high-quality HDR images, two persistent issues must be addressed.…”

“…Moreover, the demosaicing increases the computational complexity and may yield color artifacts. Second, both networks in [40], [41] are optimized with an L 2 loss, which is sensitive to large and small errors and does not correlate well to human perception [42]- [44]. Therefore, these algorithms often provide over-smoothed HDR images that are inconsistent with human visual perception, as we will discuss in Section IV.…”