Temperature‐Robust Learned Image Recovery for Shallow‐Designed Imaging Systems

Abstract

Imaging systems are widely applied in harsh environments where the performance of shallow‐designed systems may deviate from expectation. As a representative scenario, environmental temperature variation may degrade image quality due to thermal defocus and sensor response, resulting in blur and noise. However, extensive athermalization in optics usually requires a complex design process and is limited by materials. Herein, a multibranch computational imaging scheme is developed, using emerging generative adversarial networks as the postprocessing to compensate for degradation of all kinds caused by thermal defocus and noise. In addition, a temperature controllable data acquisition, division, and mixture scheme is described to facilitate effective datasets for model robustness. Experiments on a vehicle lens and a mobile phone lens reveal that the proposed multibranch learned strategy notably increases image quality in the temperature range of 0–80 °C, and outperforms conventional athermalization in most instances, which is beneficial to lowering the design and manufacturing costs of imaging systems.

Real‐Time Noise Source Estimation of a Camera System from an Image and Metadata

“…[7,29] also address camera specifics like the shutter mechanism, individual color channel biases, or differentiate between analog/digital gain. There have also been attempts to approximate noise models by DNNs [31][32][33][34] for synthesis, but [29] shows that "The DNN-based [noise generators] still cannot outperform physics-based statistical methods". All the aforementioned models calibrate their parameters (temperature, exposure time, International Organization for Standardization gain, …) offline and only implicitly account for changing camera parameters during training data generation, but they do not consider camera parameters at inference time.…”

“…[7,29] also address camera specifics like the shutter mechanism, individual color channel biases, or differentiate between analog/digital gain. There have also been attempts to approximate noise models by DNNs [ 31–34 ] for synthesis, but [ 29 ] shows that “The DNN‐based [noise generators] still cannot outperform physics‐based statistical methods”.…”