View-channel-depth light-field microscopy: real-time volumetric reconstruction of biological dynamics by deep learning

Abstract

The artefacts and non-uniform resolution accompanied with slow reconstruction speed in 13 light-field microscopy compromises its full capability for intoto observing fast biological dynamics.14 Here we demonstrate that combining a view-channel-depth (VCD) neural network with light-field 15 microscopy can mitigate these limitations, yielding artefact-free 3D image sequences with uniform 16 spatial resolution and three-orders higher, video-rate reconstruction throughput. We image neuronal 17 activities across moving C. elegans and pumping blood flow in beating zebrafish heart at single-cell 18 resolution and volume rate up to 200 Hz. 20A recurring challenge in biology is the quest to extract ever more spatiotemporal information from the targets 21 since many millisecond-transient cellular processes occur in three-dimensional tissues and across long time 22 scale. Several imaging techniques, including epifluorescence and plane illumination methods, can image live 23 samples in three dimensions at high spatial resolution 1-4 . Meanwhile, they need to record a number of 2D 24 images to comprise a 3D volume, in which the transient temporal profiles are compromised due to extended 25 acquisition time. 26The recent advent of Light field microscopy (LFM) has become a unique technique of choice for rapid and 27 instantaneous volumetric imaging 5-9 . It particularly permits the retrieval of transient 3D signal distribution 28 through post-processing of 4D light field recorded by single camera snapshot. As LFM provides high-speed 29 volumetric imaging only limited by the camera frame rate, it has delivered promising results for various 30 applications, such as recording of neuronal activities and visualization of cardiac dynamics in model 31 organisms 9-11 . Despite this advancement, its generally low spatial resolution, especially non-uniform axial 32 resolution, and presence of reconstruction artefacts by traditional light-field recovery methods yet prevents 33 its more widespread applications to capture millisecond time-scale biological processes at cellular resolution. 34Though these problems can be mitigated through optimizing the way light field being recorded 9,12 , the extra 35 system complexity could impede the wide dissemination of LFM technique. Furthermore, current LFMs 36 heavily rely on computationally demanding, iterative recovery process that intrinsically limits the overall 37 throughput of LFM reconstruction, compromising its potentials for long time-scale applications. 38Here we propose a novel LFM strategy based on a View-Channel-Depth neural network processing of light 39 field data 13 , termed VCD-LFM. By developing a light-field projection based on wave optics model, we 40 generated plenty of synthetic light-field images from high-resolution 3D images experimentally obtained 41 beforehand, and readily paired them as target and ground-truth data, respectively, for network training. The 42VCD network (VCD-Net) procedure was designed to enable the extraction of multiple views from these 2D 43 light fields...

A Hybrid of Light-Field and Light-Sheet Imaging to Decouple Myocardial Biomechanics from Intracardiac Flow Dynamics

“…Despite the sub-optimal spatial resolution for the densely organized cardiomyocytes, LFM demonstrated the capability for tracking the sparsely distributed signals like blood cells (Truong et al, 2020;Wagner et al, 2019;Wang et al, 2020). This is further applicable to study the myocardial Calcium (Ca 2+ ) flux via the genetically encoded Ca 2+ indicators (GECIs) such as GCaMP for electromechanical coupling (Weber et al, 2017) and particle tracers such as nanoparticles (Craig et al, 2012) to extend the application of our method.…”

“…1A: upper panel) (Truong et al, 2020). Our deep-learning reconstruction algorithm (Wang et al, 2020) allowed for high-throughput reconstruction of the traveling blood cells from the time-dependent light-field sequences. Using LSFM conjugated with a retrospective gating method (Lee et al, 2016;Liebling et al, 2005;Mickoleit et al, 2014;Weber et al, 2017), we performed 3-D reconstruction of the contracting myocardium ( Fig.…”

“…To establish the orthogonal geometry of LSFM, we used a dual illumination system featuring both selective volume and plane illumination for dual-channel detection. (Wang et al, 2020) Two dry objectives (Plan Fluor 4×/0.13, Nikon) for illumination were placed in the opposing direction, and were perpendicularly positioned to the water dipping detection objective (Fluor 20×/0.5w, Nikon).…”

“…We adopted a deep-learning model to provide end-to-end conversion from the 2-D light field measurements to 3-D image stacks, providing a rapid 3-D reconstruction of the dsRed-labeled cells. (Wang et al, 2020) We trained the model with the static confocal images of blood cells in the hearts and vasculatures from 16 zebrafish embryos at 3-4 dpf. Next, we enabled the recovery of a 3-D volume (−50 to 50 in depth) from the light-field raw data.…”

“…The LSFM system was capable of rapidly visualizing the embryonic zebrafish heart at 200 frames per second with high spatial resolution and minimal phototoxicity (Fei et al, 2016;Weber and Huisken, 2015), and an optical gating algorithm synchronized the irregular cardiac periodicity for performing either instantaneous or time-lapse imaging (Lee et al, 2016;Mickoleit et al, 2014;Taylor et al, 2012;Taylor et al, 2019). To track numerous traveling blood cells in space and time, we integrated LFM to image the dynamic signals at single snap shots (Truong et al, 2020;Wagner et al, 2019;Wang et al, 2020). While LSFM acquires a series of plane images across the depth of sample to reconstruct a 3-D image, LFM captures a 3-D image from single 2-D light-field detection.…”