Programmable 3D snapshot microscopy with Fourier convolutional networks

Abstract

3D snapshot microscopy enables volumetric imaging as fast as a camera allows by capturing a 3D volume in a single 2D camera image, and has found a variety of biological applications such as whole brain imaging of fast neural activity in larval zebrafish. The optimal microscope design for this optical 3D-to-2D encoding to preserve as much 3D information as possible is generally unknown and sample-dependent. Highly-programmable optical elements create new possibilities for sample-specific computational optimization of microscope parameters, e.g. tuning the collection of light for a given sample structure, especially using deep learning. This involves a differentiable simulation of light propagation through the programmable microscope and a neural network to reconstruct volumes from the microscope image. We introduce a class of global kernel Fourier convolutional neural networks which can efficiently integrate the globally mixed information encoded in a 3D snapshot image. We show in silico that our proposed global Fourier convolutional networks succeed in large field-of-view volume reconstruction and microscope parameter optimization where traditional networks fail.Preprint.

Review of data processing of functional optical microscopy for neuroscience

“…Computational imaging represents a paradigm wherein optical and algorithmic components are codesigned to compensate and enhance each other and achieve superior results to advances in either area independently. 188,189 Codesigned approaches are nascent in in-vivo functional imaging of the brain, with only a few examples aimed at faster imaging 41,190 or volumetric imaging. 16,45,191,192 The algorithmic designs for computational imaging tend to require specialized and often unique processing elements that invert the optical path of the co-designed microscope.…”

Data Processing of Functional Optical Microscopy for Neuroscience

“…Computational imaging represents a paradigm wherein optical and algorithmic components are co-designed to compensate and enhance each other and achieve superior results to advances in either area independently. 178,179 Co-designed approaches are nascent in in-vivo functional imaging of the brain, with only a few examples aimed at faster imaging 39,180 or volumetric imaging. 28 The algorithmic designs for computational imaging tend to require specialized and often unique processing elements that invert the optical path of the co-designed microscope.…”

Pupil engineering for extended depth-of-field imaging in a fluorescence miniscope

“…1(e)]. Compared to gradient-based optimization strategies, such as those based on deep learning, 25,29,30 our genetic algorithm allows for more flexible incorporation of nondifferentiable constraints, such as constraining the relative peak intensity over the desired EDoF range and binarizing the phase profile. Our algorithm is built on a linear shift invariant Fourieroptics model and, in addition, incorporates the native spherical aberration of the GRIN lens and the scattering-induced intensity decay.…”