End-to-end High Dynamic Range Camera Pipeline Optimization

Impact of ISP Tuning on Object Detection

“…Hansen et al also processed the ImageNet images with a software ISP to benchmark the classification performance against different ISP blocks and found that enabling the tone mapping ISP block increased the model accuracy by 5.8%. Following on the work completed by Mosleh et al [10], Robidoux et al [21] presented a methodology to optimize the hardware ISP of HDR cameras using an object detection KPI as the reward function, where they achieved a 33% increase in mAP and mAR over an expert-tuned ISP. They utilized their own raw ADAS dataset and the YOLOv4 [22] object detection algorithm.…”

“…Many publications make use of a reverse ISP methodology [14,20] that generates pseudo-raw data due to the lack of large-scale raw datasets. These pseudo-raw images provide useful insight into performance trends; however, the use of a truly raw dataset, as is the case with [21], is necessary to accurately represent a real camera system implementation.…”

Software-Defined Imaging: A Survey

“…The key is differentiability: making the end-to-end pipeline differentiable from image capture to vision task allow for joint optimization and the backpropagation of gradient information to update network layers. This differentiability has even extended to the realm of optical computing, allowing the codesign of optics and sensors with deep learning networks [104], [140], [141], [142].…”

Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging

“…Joint optimization, also known as end-to-end optimization or deep optics, seeks to address this by jointly optimizing optics and image processing together for either low-level imaging or high-level vision tasks. Deep optics has been applied to low-level problems, such as color imaging and demosaicking [95], extended depth of field and superresolution imaging [96], high dynamic range (HDR) imaging [97], and depth estimation [98], [99], [100], and high level problems, such as classification [101] and object detection [102]. Deep optics has also been used in time of flight imaging [103], [104], [105], computational microscopy [106], [107], [108], and imaging through scattering [109].…”

“…Robidoux et al [102] propose an alternative method for hardware ISP optimization in the framework of end-to-end optimization of multi-exposure high dynamic range (HDR) camera systems. In their work, optimization alternates between 0 th -order evolutionary search over sensor and ISP parameters and 1 st -order gradient descent on the neural network weights, where the neural network is trained for a perception task, such as automotive object detection, using images captured by the camera system.…”