Single image HDR reconstruction using a CNN with masked features and perceptual loss

Abstract

Digital cameras can only capture a limited range of real-world scenes' luminance, producing images with saturated pixels. Existing single image high dynamic range (HDR) reconstruction methods attempt to expand the range of luminance, but are not able to hallucinate plausible textures, producing results with artifacts in the saturated areas. In this paper, we present a novel learning-based approach to reconstruct an HDR image by recovering the saturated pixels of an input LDR image in a visually pleasing way. Previous deep learning-based methods apply the same convolutional filters on wellexposed and saturated pixels, creating ambiguity during training and leading to checkerboard and halo artifacts. To overcome this problem, we propose a feature masking mechanism that reduces the contribution of the features from the saturated areas. Moreover, we adapt the VGG-based perceptual loss function to our application to be able to synthesize visually pleasing textures. Since the number of HDR images for training is limited, we propose to train our system in two stages. Specifically, we first train our system on a large number of images for image inpainting task and then fine-tune it on HDR reconstruction. Since most of the HDR examples contain smooth regions that are simple to reconstruct, we propose a sampling strategy to select challenging training patches during the HDR fine-tuning stage. We demonstrate through experimental results that our approach can reconstruct visually pleasing HDR results, better than the current state of the art on a wide range of scenes.

Deep Tone-Mapping Operator Using Image Quality Assessment Inspired Semi-Supervised Learning

“…Jang et al [40] introduced ΔE76 color difference formula in their loss function, which was the first human perception related loss term in HDR related CNN. Santos et al [43] first introduced gram matrix [44] to measure style loss, and feature masking to emphasize global difference.…”

“…Hence, an extra task of strengthening the global comprehension of Full-scale Local Branch NL was posed. Other methods tackled this by introducing multi-group (in [56]) or multi-pass (in [8]) convolution where different kernel-size are assigned to each pass, using spatial attention mechanism (in [43], [50] and [5]), using 1-D and 2-D dynamic convolution (in [47]), and improving encoder-decoder structure (U-net) [57].…”

“…Perceptual loss (supervised). Loss from another pre-trained network was referred to as "perceptual" loss in 6 of 20 related works ( [3], [25], [27], [42], [43] and [46]), we decided to follow the same practice. By separately feeding S, SD or SB and SL, SLD or SLB into pretrained 19-layer VGG-Net [30], perceptual loss (lP) was calculated using the mean absolute error between same VGG layers with different inputs.…”

“…Since there is a huge difference between pixel value distribution of linear HDR and non-linear SDR images, majority of related works followed the common practice of converting HDR images into logarithm (log) domain ( [3], [33], [25], [26], [34], [36], [42], [43] and [46]) or other non-linear domain ( [38]), which made pixel value more evenlydistributed and easier for CNN to process. While the rest of methods ( [31], [24], [27], [39], [40], [41], [45], [5], [47] and [56]) just normalized HDR images without domain transfer.…”

Single-Shot High Dynamic Range Imaging via Multiscale Convolutional Neural Network

“…Although each algorithm addresses an important issue, no algorithm has enough robustness to completely handle the misalignment caused by object motion in images. Concurrently, another approach attempts to infer an HDR image directly from a single LDR image using CNNs [26]- [31], which is also known as inverse tonemapping (ITM). Although the networks in this approach can recover missing details in under-and over-exposed regions, one limitation of this approach is its high dependence on a single input LDR image, thereby lacking underlying information.…”