Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning

Abstract

Light-field microscopy has emerged as a technique of choice for high-speed volumetric imaging of fast biological processes. However, artefacts, non-uniform resolution, and a slow reconstruction speed have limited its full capabilities for in toto extraction of the dynamic spatiotemporal patterns in samples. Here, we combined a view-channel-depth (VCD) neural network with light-field microscopy to mitigate these limitations, yielding artefact-free three-dimensional image sequences with uniform spatial resolution and high video-rate reconstruction throughput. We imaged neuronal activities across moving C. elegans and blood flow in a beating zebrafish heart at single-cell resolution with volumetric imaging rates up to 200 Hz.

A hybrid of light-field and light-sheet imaging to study myocardial function and intracardiac blood flow during zebrafish development

“…The heart was selectively illuminated by a rod-shaped laser beam to eliminate background noise and to enhance image contrast ( Fig 1A : upper panel ) [ 30 ]. We adopted a deep-learning algorithm [ 31 ] for 3-D reconstruction of the blood cells acquired from the raw 2-D light-field sequences. This algorithm reconstructed an equivalent 3-D imaging speed at 200 vps.…”

“…Data were collected at the frame rate of 200 Hz (at a cropped frame size: 768 × 768 px) and exposure of 5 ms for all imaging tasks. We adopted a deep-learning model to provide end-to-end conversion from the 2-D light field measurements to 3-D image stacks, providing a rapid 3-D reconstruction of the blood cells [31]. We trained the model with the static confocal images of blood cells in the hearts and vasculatures from 16 zebrafish embryos at 3-4 dpf.…”

“…Unlike the conventional scanning imaging, the LFM captures both the lateral and angular events from the incident light in a single snapshot; thus, enabling instantaneous volumetric imaging via post-processing [27,28]. LFM has demonstrated the capacity to capture the sparse dynamic signals such as red blood cells [29][30][31]. Using transgenic zebrafish labeling cardiac myosin light chain, cmlc2, with GFP and flowing blood cells, gata1a, with dsRed, we were able to capture the myocardial displacement, along with intracardiac blood flow at 200 volumes per second (vps, defined as the number of acquired stacks of 2-D cross-sections of the sample per second) from 3 to 5 days post fertilization (dpf).…”

Automatic Segmentation and Cardiac Mechanics Analysis of Evolving Zebrafish Using Deep Learning

“…Previously, segmentation of the LSFM images for measuring cardiac mechanics was accomplished manually by recognizing different intensities from large amounts of samples or tissue scatterings engenders many variables ( 18 ). The time-consuming task of manually segmenting the LSFM images is infeasible when processing high axial resolution data, as the number of images required is enormous ( 19 , 20 ). On the other hand, lower axial resolution degrades the volume measurement's accuracy, while inconsistent manual segmentation poses a threat to the cardiac mechanic analysis' overall quality.…”

Deep-learning on-chip DSLM enabling video-rate volumetric imaging of neural activities in moving biological specimens

“…9,33,34 The recurrent concern on how to improve the speed without sacrificing the accuracy is previously addressed by single-image super-resolution (SISR) approaches, such as dictionary search and compressed sensing, but only limited to certain types of spare signals. 35,36 The recent advent of deep learning-enabled image restoration has brought big impact to the microscopy field, 22,[37][38][39][40][41] with the trained neural network capable of directly deducing a higher-quality image based on a single low-quality measurement. Deep learning-based restoration has been also applied to the denoising of microscopy images, in which the acquisition of qualitied microscopy data for network training is relatively difficult.…”