Learning to Reconstruct Confocal Microscopy Stacks from Single Light Field Images

Abstract

We present a novel deep learning approach to reconstruct confocal microscopy stacks from single light field images. To perform the reconstruction, we introduce the LFMNet, a novel neural network architecture inspired by the U-Net design [1]. It is able to reconstruct with high-accuracy a 112 × 112 × 57.6µm 3 volume (1287 × 1287 × 64 voxels) in 50ms given a single light field image of 1287 × 1287 pixels, thus dramatically reducing 720-fold the time for confocal scanning of assays at the same volumetric resolution and 64-fold the required storage. To prove the applicability in life sciences, our approach is evaluated both quantitatively and qualitatively on mouse brain slices with fluorescently labelled blood vessels. Because of the drastic reduction in scan time and storage space, our setup and method are directly applicable to real-time in vivo 3D microscopy. We provide analysis of the optical design, of the network architecture and of our training procedure to optimally reconstruct volumes for a given target depth range. To train our network, we built a data set of 362 light field images of mouse brain blood vessels and the corresponding aligned set of 3D confocal scans, which we use as ground truth. The data set will be made available for research purposes [2].

Reconstruct high-resolution multi-focal plane images from a single 2D wide field image

“…With the progress of computational imaging, deep learning based 2D-to-3D super-resolution methods are gradually emerging. Page et al [30] proposed LFMNet to reconstruct confocal microscopy stacks from single light filed images. Wu et al [17] proposed Deep-Z model based on GAN, which can refocus single 2D wide-field fluorescence images to obtain 3D images.…”

Volumetric segmentation for integral microscopy with Fourier plane recording

“…In this work we aim at describing a specimen volume through computationally segmenting the ZStack into object pixels of interest, making use of our previous work on refocusing [5]. The possibility of fully reconstructing a volume from an LF microscope image has been investigated in [10] and [11]. Both works utilize single-shot LF microscope images with high angular resolution.…”