Multi-element microscope optimization by a learned sensing network with composite physical layers

Kim, Kanghyun; Konda, Pavan Chandra; Cooke, Colin; Appel, Ron D.; Horstmeyer, Roarke

doi:10.1364/ol.401105

Cited by 14 publications

(7 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Rather than following the static optical setup of the microscope and postprocessing its acquired images, recent methods have focused on optimizing certain parts of the optical hardware itself. Several approaches focus on optimizing the illumination patterns of the microscope [5,13,17,27]. This research direction of jointly optimizing the forward optics (by learning illumination patterns) with the inverse reconstruction model has been able to reduce the data requirement in QPM [15,16].…”

Section: Related Workmentioning

confidence: 99%

“…Despite the significant performance boost in such methods, the limitations inherited from the hardware of the microscope set the upper performance bound [7]. Therefore, over the past decade, researchers have focused on joint deep-learning optimization of not only the reconstruction model but also the hardware of the microscope itself [3,5,13,17,27,36]. Nevertheless, all these methods' focus was to optimize a system that is already capable of a specific imaging task.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Differentiable Microscopy Designs an All Optical Quantitative Phase Microscope

Herath¹,

Haputhanthri²,

Hettiarachchi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Ever since the first microscope by Zacharias Janssen in the late 16th century, scientists have been inventing new types of microscopes for various tasks. Inventing a novel architecture demands years, if not decades, worth of scientific experience and creativity. In this work, we introduce Differentiable Microscopy (∂µ), a deep learning-based design paradigm, to aid scientists design new interpretable microscope architectures. Differentiable microscopy first models a common physics-based optical system however with trainable optical elements at key locations on the optical path. Using pre-acquired data, we then train the model end-to-end for a task of interest. The learnt design proposal can then be simplified by interpreting the learnt optical elements. As a first demonstration, based on the optical 4-f system, we present an all-optical quantitative phase microscope (QPM) design that requires no computational post-reconstruction. A follow-up literature survey suggested that the learnt architecture is similar to the generalized phase concept developed two decades ago. We then incorporate the generalized phase contrast concept to simplify the learning procedure. Furthermore, this physical optical setup is miniaturized using a diffractive deep neural network (D2NN). We outperform the existing benchmark for all-optical phase-to-intensity conversion on multiple datasets, and ours is the first demonstration of its kind on D2NNs. The proposed differentiable microscopy framework supplements the creative process of designing new optical systems and would perhaps lead to unconventional but better optical designs.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Differentiable Microscopy Designs an All Optical Quantitative Phase Microscope

Herath¹,

Haputhanthri²,

Hettiarachchi³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…See [ 91 , 92 ] for further details. The idea of learned sensing was also applied in the optical domain in order to determine optimal illumination patterns for specific microscopy tasks [ 93 ].…”

Section: Related Workmentioning

confidence: 99%

A Baseline for Cross-Database 3D Human Pose Estimation

Rapczyński

Werner

Handrich

et al. 2021

Sensors

View full text Add to dashboard Cite

Vision-based 3D human pose estimation approaches are typically evaluated on datasets that are limited in diversity regarding many factors, e.g., subjects, poses, cameras, and lighting. However, for real-life applications, it would be desirable to create systems that work under arbitrary conditions (“in-the-wild”). To advance towards this goal, we investigated the commonly used datasets HumanEva-I, Human3.6M, and Panoptic Studio, discussed their biases (that is, their limitations in diversity), and illustrated them in cross-database experiments (for which we used a surrogate for roughly estimating in-the-wild performance). For this purpose, we first harmonized the differing skeleton joint definitions of the datasets, reducing the biases and systematic test errors in cross-database experiments. We further proposed a scale normalization method that significantly improved generalization across camera viewpoints, subjects, and datasets. In additional experiments, we investigated the effect of using more or less cameras, training with multiple datasets, applying a proposed anatomy-based pose validation step, and using OpenPose as the basis for the 3D pose estimation. The experimental results showed the usefulness of the joint harmonization, of the scale normalization, and of augmenting virtual cameras to significantly improve cross-database and in-database generalization. At the same time, the experiments showed that there were dataset biases that could not be compensated and call for new datasets covering more diversity. We discussed our results and promising directions for future work.

show abstract

“…Despite the significant performance boost in such methods, the limitations inherited from the hardware of the microscope set the upper performance bound [5]. Therefore, over the past decade, researchers have focused on joint deep-learning optimization of not only the reconstruction model but also the hardware of the microscope itself [6][7][8][9][10][11]. Nevertheless, all these methods' focus was to optimize a system that is already capable of a specific imaging task.…”

Section: Introductionmentioning

confidence: 99%

Differentiable Microscopy Designs an All Optical Quantitative Phase Microscope

Herath

Haputhanthri

Hettiarachchi

et al. 2022

Preprint

View full text Add to dashboard Cite

Ever since the first microscope by Zacharias Janssen in the late 16th century, scientists have been inventing new types of microscopes for various tasks. Inventing a novel architecture demands years, if not decades, worth of scientific experience and creativity. In this work, we introduce Differentiable Microscopy (∂μ), a deep learning-based design paradigm, to aid scientists design new interpretable microscope architectures. Differentiable microscopy first models a common physics-based optical system however with trainable optical elements at key locations on the optical path. Using pre-acquired data, we then train the model end-to-end for a task of interest. The learnt design proposal can then be simplified by interpreting the learnt optical elements. As a first demonstration, based on the optical 4-f system, we present an all-optical quantitative phase microscope (QPM) design that requires no computational post-reconstruction. A follow-up literature survey suggested that the learnt architecture is similar to the generalized phase concept developed two decades ago. We then incorporate the generalized phase contrast concept to simplify the learning procedure. Furthermore, we also demonstrate that similar results can be achieved by an uninterpretable learning based method, namely diffractive deep neural networks (D2NN). We outperform the existing benchmark for all-optical phase-to-intensity conversion on multiple datasets, and ours is the first demonstration of its kind on D2NNs. The proposed differentiable microscopy framework supplements the creative process of designing new optical systems and would perhaps lead to unconventional but better optical designs.

show abstract

Multi-element microscope optimization by a learned sensing network with composite physical layers

Cited by 14 publications

References 10 publications

Differentiable Microscopy Designs an All Optical Quantitative Phase Microscope

Differentiable Microscopy Designs an All Optical Quantitative Phase Microscope

A Baseline for Cross-Database 3D Human Pose Estimation

Differentiable Microscopy Designs an All Optical Quantitative Phase Microscope

Contact Info

Product

Resources

About