Fluorescence lifetime image microscopy prediction with convolutional neural networks for cell detection and classification in tissues

Abstract

Convolutional neural networks (CNNs) and other deep-learning models have proven to be transformative tools for the automated analysis of microscopy images, particularly in the domain of cellular and tissue imaging. These computer-vision models have primarily been applied with traditional microscopy image modalities (e.g. brightfield and fluorescence), likely due to the availability of large datasets in these regimes. However, more advanced microscopy imaging techniques could, potentially, allow for improved model performance in various computational histopathology tasks. In this work, we demonstrate that CNNs can achieve high accuracy in cell detection and classification without large amounts of data when applied to histology images acquired by FLIM. This accuracy is higher than what would be achieved with regular single or dual-channel fluorescence images under the same settings, particularly for CNNs pretrained on publicly available fluorescent cell or general photographic datasets. Additionally, generated FLIM images could be predicted from just the fluorescence image data by using a dense U-Net CNN model trained on a subset of ground-truth FLIM images. These U-Net CNN generated FLIM images demonstrated high similarity to ground truth and improved accuracy in cell detection and classification over fluorescence alone when used as input to a variety of commonly used CNNs. This improved accuracy was maintained even when the FLIM images were generated by a U-Net CNN trained on only a few example FLIM images.

Identifying lipid particle sub-types in live Caenorhabditis elegans with two-photon fluorescence lifetime imaging

“…Our measurement protocol and analysis pipeline can be directly applied to other FLIM instruments such as frequency-domain FLIM ( Raspe et al, 2016 ; Dunkers et al, 2021 ) or real-time pixel phasor displayed FLIM ( Sorrells et al, 2021 ). Applying advanced deep learning algorithms such as U-Net convolutional neural network (CNN) ( Smolen and Wooley, 2022 ) could possibly further improve the specificity and accuracy of Nile Red 2p-FLIM. Since the two-photon laser can simultaneously excite multiple dyes or fluorescent proteins, it is possible to study lipid/yolk interactions with other labeled proteins or organelles in live C. elegans .…”

Probing organoid metabolism using fluorescence lifetime imaging microscopy (FLIM): The next frontier of drug discovery and disease understanding

“…Recent advances in computing and imaging systems have created a new research dimension with large datasets from single experiments, increasing the complexity of the data with higher numbers of conditions and dependent variables [257][258][259]. FLIM can benefit from ML with improved analysis and calculation of fluorescence lifetime (ideally addressing the photon budget issues), clustering and segmenting imaged species, help with tagging when combined with other analytical methods, or even to predict the data [25,58,84,260,261].…”

“…Notable and very recent mentions include FLI-Net [84]; a deep learning neural network using deep learning, to quantify fluorescence decays simultaneously over a whole image and at fast speeds and flimGANE [270] to profile cells using deep learning and generate accurate and high-quality FLIM images even in the photon-starved conditions. Furthermore; another extension of this AI subset; convolutional neural networks (CNN) have demonstrated applicability using FLIM data and micrographs with accurate cell tracking and classification performed [261]; and have been able to predict FLIM images based on fluorescent data when trained using a small subset of FLIM images; with a high degree of accuracy. Such applications are rapidly gathering momentum and are at the cutting edge of the field [97,271,272].…”