dO: A Differentiable Engine for Deep Lens Design of Computational Imaging Systems

Abstract

Computational imaging systems algorithmically post-process acquisition images either to reveal physical quantities of interest or to increase image quality, e.g., deblurring. Designing a computational imaging system requires co-design of optics and algorithms, and recently Deep Lens systems have been proposed in which both components are end-to-end designed using data-driven end-to-end training. However, progress on this exciting concept has so far been hampered by the lack of differentiable forward simulations for complex optical design spaces.Here, we introduce dO (DiffOptics) to provide derivative insights into the design pipeline to chain variable parameters and their gradients to an error metric through differential ray tracing. However, straightforward back-propagation of many millions of rays requires unaffordable device memory, and is not resolved by prior works. dO alleviates this issue using two customized memory-efficient techniques: differentiable raysurface intersection and adjoint back-propagation. Broad application examples demonstrate the versatility and flexibility of dO, including classical lens designs in asphere, double-Gauss, and freeform, reverse engineering for metrology, and joint designs of optics-network in computational imaging applications. We believe dO enables a radically new approach to computational imaging system designs and relevant research domains.

Curriculum Learning for ab initio Deep Learned Refractive Optics

“…The DeepLens optimization uses differentiable raytracing [26,31,34] as an optical simulator. Briefly, the core concept of differentiable ray-tracing is to automatically track derivative information while the calculations of a classical ray-tracing simulation.…”

“…Our work builds on top of the dO engine [31], which provides a memory-efficient differentiable ray-tracing framework that enables complex design tasks on desktop computers. Please see [31] and Supplemental Document 1 for more details on the basic differentiable ray-tracing operations.…”

“…Deep lens design has recently emerged as a promising new paradigm for jointly optimizing optical designs and downstream image reconstruction methods [23,32,25,26,31,29]. A deep lens framework is powered by differentiable optical simulators and optimization based on error back-propagation (or reverse mode auto-differentiation) in combination with error metrics that directly measure final reconstructed image quality rather than classical and manually tuned figures of merit.…”

“…Most recently, there has been an effort to expand the deep lens design paradigm to compound optical systems composed of multiple refractive optical elements [26,8,31,6,29,34]. The core methodology behind these efforts is optical simulation based on differentiable ray-tracing, in which the evolution of image quality can be tracked as a function of design parameters such as lens curvatures or placements of lens elements.…”

Automatic Lens Design based on Differentiable Ray-tracing

“…It's typically realized by experienced engineers with constant interventions during the optimization. Recent work in automatic staring point generation [1] and differentiable ray-tracing [2][3][4][5] show a promising potential to solve optical design problems as an optimization task with neural networks. But they either can only generate spheric lenses or need a well-designed starting point.…”