Deep learning approach for Fourier ptychography microscopy

Abstract

Convolutional neural networks (CNNs) have gained tremendous success in solving complex inverse problems. The aim of this work is to develop a novel CNN framework to reconstruct video sequence of dynamic live cells captured using a computational microscopy technique, Fourier ptychographic microscopy (FPM). The unique feature of the FPM is its capability to reconstruct images with both wide field-of-view (FOV) and high resolution, i.e. a large space-bandwidth-product (SBP), by taking a series of low resolution intensity images. For live cell imaging, a single FPM frame contains thousands of cell samples with different morphological features. Our idea is to fully exploit the statistical information provided by this large spatial ensembles so as to make predictions in a sequential measurement, without using any additional temporal dataset. Specifically, we show that it is possible to reconstruct high-SBP dynamic cell videos by a CNN trained only on the first FPM dataset captured at the beginning of a time-series experiment. Our CNN approach reconstructs a 12800×10800 pixels phase image using only ∼25 seconds, a 50× speedup compared to the model-based FPM algorithm. In addition, the CNN further reduces the required number of images in each time frame by ∼ 6×. Overall, this significantly improves the imaging throughput by reducing both the acquisition and computational times. The proposed CNN is based on the conditional generative adversarial network (cGAN) framework. We further propose a mixed loss function that combines the standard image domain loss and a weighted Fourier domain loss, which leads to improved reconstruction of the high frequency information. Additionally, we also exploit transfer learning so that our pre-trained CNN can be further optimized to image other cell types. Our technique demonstrates a promising deep learning approach to continuously monitor large live-cell populations over an extended time and gather useful spatial and temporal information with sub-cellular resolution.

Deep phase decoder: self-calibrating phase microscopy with an untrained deep neural network

“…More recently, deep neural networks (DNNs), typically trained in an end-toend fashion on large datasets to directly map given intensities back to phase, have been used to obtain efficient phase retrieval algorithms. For phase microscopy, trained DNNs give state-of-the-art performance in holographic [18], lensless [19], ptychographic [20], and through-scattering-media [21,22] configurations, among others [23]. The results have validated the efficiency of properly trained DNNs to solve non-linear inverse problems and shifted the computational paradigm in QPM towards predominantly data-driven frameworks.…”

Does deep learning always outperform simple linear regression in optical imaging?

“…In recent years, deep learning receives much attention in many research fields including optical design [1,2] and optical imaging [3]. In previous works, deep learning has been extensively applied for many optical imaging problems including phase retrieval [4][5][6][7], microscopic image enhancement [8][9], scattering imaging [10][11], holography [12][13][14][15][16][17][18], single-pixel imaging [19,20], super-resolution [21][22][23][24], Fourier ptychography [25][26][27], optical interferometry [28,29], wavefront sensing [30,31], and optical fiber communications [32].…”

Learning to synthesize: robust phase retrieval at low photon counts

“…Recently, deep learning regression has been investigated for application to nonlinear inverse problems, in particular phase retrieval: direct [37][38][39], holographic [40,41], and ptychographic [42,43]. The idea, described briefly in Section 2.B, is to train a deep neural network (DNN) in supervised mode from examples of phase objects and their intensity images so that, after training, given an intensity image as input, the DNN outputs an estimate of the phase object.…”