End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging

Sitzmann, Vincent; Diamond, Steven; Peng, Yifan; Xiong, Deping; Boyd, Stephen; Heidrich, Wolfgang; Heide, Felix; Wetzstein, Gordon

doi:10.1145/3197517.3201333

Cited by 332 publications

(198 citation statements)

References 53 publications

Supporting

Mentioning

196

Contrasting

Unclassified

Order By: Relevance

“…Deep Optics Deep learning can be used for jointly training camera optics and CNN-based estimation methods. This approach was recently demonstrated for applications in extended depth of field and superresolution imaging [39], image classification [2], and multicolor localization microscopy [25]. For example, Hershko et al [25] proposed to learn a custom diffractive phase mask that produced highly wavelength-dependent point spread functions (PSFs), allowing for color recovery from a grayscale camera.…”

Section: Computational Photography For Depth Estimationmentioning

confidence: 99%

“…Inspired by recent work on deep optics [2,39,12], we interpret the monocular depth estimation problem with coded defocus blur as an optical-encoder, electronic-decoder system that can be trained in an end-to-end manner. Although co-designing optics and image processing is a core idea in computational photography, only differentiable estimation algorithms, such as neural networks, allow for true end-toend computational camera designs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep Optics for Monocular Depth Estimation and 3D Object Detection

Chang

Wetzstein

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

169

112

View full text Add to dashboard Cite

Depth estimation and 3D object detection are critical for scene understanding but remain challenging to perform with a single image due to the loss of 3D information during image capture. Recent models using deep neural networks have improved monocular depth estimation performance, but there is still difficulty in predicting absolute depth and generalizing outside a standard dataset. Here we introduce the paradigm of deep optics, i.e. end-to-end design of optics and image processing, to the monocular depth estimation problem, using coded defocus blur as an additional depth cue to be decoded by a neural network. We evaluate several optical coding strategies along with an end-to-end optimization scheme for depth estimation on three datasets, including NYU Depth v2 and KITTI. We find an optimized freeform lens design yields the best results, but chromatic aberration from a singlet lens offers significantly improved performance as well. We build a physical prototype and validate that chromatic aberrations improve depth estimation on real-world results. In addition, we train object detection networks on the KITTI dataset and show that the lens optimized for depth estimation also results in improved 3D object detection performance.

show abstract

Section: Computational Photography For Depth Estimationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Deep Optics for Monocular Depth Estimation and 3D Object Detection

Chang

Wetzstein

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

169

112

View full text Add to dashboard Cite

show abstract

“…There are several recent works that consider use of machine learning to jointly optimize hardware and software for imaging tasks [5,6,7,8,9,10,11]. These approaches aim to find a fixed set of optical parameters that are optimal for a particular task.…”

Section: Previous Workmentioning

confidence: 99%

Towards an Intelligent Microscope: Adaptively Learned Illumination for Optimal Sample Classification

Chaware

Cooke

Kim

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Recent machine learning techniques have dramatically changed how we process digital images. However, the way in which we capture images is still largely driven by human intuition and experience. This restriction is in part due to the many available degrees of freedom that alter the image acquisition process (lens focus, exposure, filtering, etc). Here we focus on one such degree of freedom -illumination within a microscope -which can drastically alter information captured by the image sensor. We present a reinforcement learning system that adaptively explores optimal patterns to illuminate specimens for immediate classification. The agent uses a recurrent latent space to encode a large set of variablyilluminated samples and illumination patterns. We train our agent using a reward that balances classification confidence with image acquisition cost. By synthesizing knowledge over multiple snapshots, the agent can classify on the basis of all previous images with higher accuracy than from naively illuminated images, thus demonstrating a smarter way to physically capture task-specific information.

show abstract

“…These methods require a huge number of training examples to properly learn millions of parameters that model the reconstruction process, and they often do not transfer well to the experimental setting. In comparison, model-based methods [14], [15] are able to efficiently learn the experimental design with very few training examples and have been shown to learn designs that do transfer well to the experimental setting. This is achieved by unrolling the image reconstruction process [16]- [19] into a Physics-based Neural Network (PbNN) [14] and then learning the experimental design parameters to maximize system performance.…”

Section: Introductionmentioning

confidence: 99%

Data-Driven Design for Fourier Ptychographic Microscopy

Kellman

Bostan

Chen

et al. 2019

2019 IEEE International Conference on Computational Photography (ICCP)

View full text Add to dashboard Cite

Fourier Ptychographic Microscopy (FPM) is a computational imaging method that is able to super-resolve features beyond the diffraction-limit set by the objective lens of a traditional microscope. This is accomplished by using synthetic aperture and phase retrieval algorithms to combine many measurements captured by an LED array microscope with programmable source patterns. FPM provides simultaneous large field-of-view and high resolution imaging, but at the cost of reduced temporal resolution, thereby limiting live cell applications. In this work, we learn LED source pattern designs that compress the many required measurements into only a few, with negligible loss in reconstruction quality or resolution. This is accomplished by recasting the super-resolution reconstruction as a Physics-based Neural Network and learning the experimental design to optimize the network's overall performance. Specifically, we learn LED patterns for different applications (e.g. amplitude contrast and quantitative phase imaging) and show that the designs we learn through simulation generalize well in the experimental setting. Further, we discuss a context-specific loss function, practical memory limitations, and interpretability of our learned designs.

show abstract

End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging

Cited by 332 publications

References 53 publications

Deep Optics for Monocular Depth Estimation and 3D Object Detection

Deep Optics for Monocular Depth Estimation and 3D Object Detection

Towards an Intelligent Microscope: Adaptively Learned Illumination for Optimal Sample Classification

Data-Driven Design for Fourier Ptychographic Microscopy

Contact Info

Product

Resources

About