Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning

Abstract

Three-dimensional (3D) fluorescence microscopy in general requires axial scanning to capture images of a sample at different planes. Here we demonstrate that a deep convolutional neural network can be trained to virtually refocus a 2D fluorescence image onto user-defined 3D surfaces within the sample volume. With this data-driven computational microscopy framework, we imaged the neuron activity of a Caenorhabditis elegans worm in 3D using a time-sequence of fluorescence images acquired at a single focal plane, digitally increasing the depth-of-field of the microscope by 20-fold without any axial scanning, additional hardware, or a trade-off of imaging resolution or speed. Furthermore, we demonstrate that this learning-based approach can correct for sample drift, tilt, and other image aberrations, all digitally performed after the acquisition of a single fluorescence image. This unique framework also cross-connects different imaging modalities to each other, enabling 3D refocusing of a single wide-field fluorescence image to match confocal microscopy images acquired at different sample planes. This deep learning-based 3D image refocusing method might be transformative for imaging and tracking of 3D biological samples, especially over extended periods of time, mitigating photo-toxicity, sample drift, aberration and defocusing related challenges associated with standard 3D fluorescence microscopy techniques. Text:Three-dimensional (3D) fluorescence microscopic imaging is essential for biomedical and physical sciences as well as engineering, covering various applications 1-7 . Despite its broad

Interpretable deep learning of label-free live cell images uncovers functional hallmarks of highly-metastatic melanoma

“…Morphology has long been a cue for cell biologists and pathologists to recognize cell type and abnormalities related to disease (Bakal et al, 2007;Chan, 2014;Eddy et al, 2018;Gordonov et al, 2015;Gurcan et al, 2009;López, 2013;Pavillon et al, 2018;Wu et al, 2020;Yin et al, 2013). In this study, we rely on the exquisite sensitivity of deep learned artificial neural networks in recognizing subtle but systematic image patterns to classify different cell types and cell states.…”

“…However, the often cited weakness of these techniques is the lack of an intuitive explanation of which parts of the data are particularly meaningful in defining the extracted pattern. While in some applications, such as image segmentation, image restoration or mapping between imaging modalities, a well-validated outcome of a network has been satisfactory (Christiansen et al, 2018;Fang et al, 2019b;Guo et al, 2019;Hershko et al, 2019;Hollandi et al, 2019;LaChance and Cohen, 2020;Moen et al, 2019;Nehme et al, 2018;Ounkomol et al, 2018;Ouyang et al, 2018;Rivenson et al, 2019;Wang et al, 2019;Weigert et al, 2018;Wu et al, 2019), there is increasing mistrust in results produced by 'black-box' neural networks. Aside from increasing the confidence, the analysis of the properties -also referred to as 'mechanisms'of the pattern recognition process can potentially generate insight of a biological/physical phenomenon that escapes the analysis driven by human intuition.…”

“…Whether morphological cues are also informative of the broader spectrum of cell signaling programs that drive hallmark functions in metastatic cells such as shifts in the regulation of metabolism, cell cycle progression, or cell death is less clear, although very recent work, using conventional shape-based machine learning of fluorescently labeled cell lines, suggests this may be the case (Wu et al, 2020).…”

Multifocal imaging for precise, label-free tracking of fast biological processes in 3D

“…While we have concentrated here on label-free samples, our method has the potential to be adapted to fluorescently-labelled samples and to be combined with state-of-art computational techniques, such as deep learning. For instance, a deep-learning method for deducing zpositions from fluorescence microscopy images was recently described 44 . Our method could incorporate similar computational techniques to make use of the information provided by multiple planes for extending the z-range and enhancing the precision in fluorescence microscopy.…”

Noise-free quantitative phase imaging in Gabor holography with conditional generative adversarial network

“…Convolutional neural network (CNN) and deep learning approaches have been proposed for several optical applications. Examples include virtual staining of non-stained samples [33], increasing spatial resolution in a large field of view in optical microscopy [34,35], color holographic microscopy with CNN [36], autofocusing and enhancing the depth-of-filed in inline holography [37], lens-less computational imaging by deep learning [38], single-cell-based reconstruction distance estimation by a regression CNN model [39], super-resolution fringe patterns by deep learning holography [40], virtual refocusing in fluorescence microscopy to map 2D images to a 3D surface [41], and several other studies [42][43][44]. Deep-learning based phase recovery by residual CNN model was also suggested [45], but the application is limited because the reference noise-free phase images for deep-learning model are generated by the multi-height phase retrieval approach (8 holograms are recorded at different sample-to-sensor distances).…”