Learning Rank-1 Diffractive Optics for Single-Shot High Dynamic Range Imaging

Abstract

High-dynamic range (HDR) imaging is an essential imaging modality for a wide range of applications in uncontrolled environments, including autonomous driving, robotics, and mobile phone cameras. However, existing HDR techniques in commodity devices struggle with dynamic scenes due to multi-shot acquisition and postprocessing time, e.g. mobile phone burst photography, making such approaches unsuitable for real-time applications. In this work, we propose a method for snapshot HDR imaging by learning an optical HDR encoding in a single image which maps saturated highlights into neighboring unsaturated areas using a diffractive optical element (DOE). We propose a novel rank-1 parameterization of the DOE which drastically reduces the optical search space while allowing us to efficiently encode high-frequency detail. We propose a reconstruction network tailored to this rank-1 parametrization for the recovery of clipped information from the encoded measurements. The proposed end-to-end framework is validated through simulation and real-world experiments and improves the PSNR by more than 7 dB over state-ofthe-art end-to-end designs.

End-to-end complex lens design with differentiate ray tracing

“…End-to-end Optics Design. Co-designing of optics and post-processing has demonstrated superior performance over traditional heuristic approaches in single-lens color imaging [Chakrabarti 2016;Peng et al 2019], HDR imaging [Metzler et al 2020Sun et al 2020a], single image depth estimation [Boominathan et al 2020;Chang and Wetzstein 2019a;Haim et al 2018;Kotwal et al 2020;Wu et al 2019a,b;Wu et al 2020;Zhang et al 2018], microscopy imaging [Horstmeyer et al 2017;Kellman et al 2019;Nehme et al 2019;Shechtman et al 2016].…”

“…A solution to this problem has been to utilize a commercial lens but add a single, co-designed element for a specific purpose. This approach has been applied to superresolution SPAD cameras [Sun et al 2020b] and high dynamic range imaging [Sun et al 2020a]. However, none of these approaches can deal with large FOVs as their image formation model relies on simple paraxial approximation in addition to the single-surface restriction.…”

“…Over the past few years, co-design of optics and image processing [Peng et al 2019;Sun et al 2018], or even data-driven end-to-end design [Sitzmann et al 2018] have emerged to bridge the gap between optical design and algorithm development. Codesign of optics and post-processing algorithms has achieved a superior performance for domain specific tasks such as depth estimation [Chang and Wetzstein 2019a], large field-of-view imaging [Peng et al 2019], extended depth-of-field [Chang and Wetzstein 2019b], optimal sampling [Sun et al 2020b], and high dynamic range (HDR) imaging [Metzler et al 2020;Sun et al 2020a]. Unfortunately, the differentiable lens models used in these works have been too limited to describe complex optical assemblies, and have instead only allowed to optimize a single optical surface with a single material.…”

18‐4: Fast Image Restoration and Glare Removal for Under‐Display Camera by Guided Filter

“…One-shot HDR reconstruction usually uses an optical filter to produce glare on purpose and utilize information remaining on tails of glares to recover the saturated regions due to strong light [3]. An end-toend optimization has been also proposed that optimizes the filter shape simultaneously using a neural network [4,5]. In these methods, the point spread function (PSF) determined by the shape of the filter is essential to recover, but the panel structure of UDC is fixed.…”

Compressive HDR Light Field Imaging Using a Single Multi-ISO Sensor

“…a pyramidshaped mirror or a refracting prism, and redirected toward a set of sensors associated with absorptive filters to produce images with different exposures. In [35], an optical design is proposed with a rank-1 phase pattern, where the diffractive optic point spread function (PSF) is jointly optimized with the reconstruction method that recovers the latent HDR scene from the input measurement. A joint design of a spatially-varying modulation mask and of the HDR image reconstruction network is proposed in [20].…”