A versatile deep learning architecture for classification and label-free prediction of hyperspectral images

Abstract

Hyperspectral imaging is a technique that provides rich chemical or compositional information not regularly available to traditional imaging modalities such as intensity imaging or color imaging based on the reflection, transmission, or emission of light. Analysis of hyperspectral imaging often relies on machine learning methods to extract information. Here, we present a new flexible architecture, the U-within-U-Net, that can perform classification, segmentation, and prediction of orthogonal imaging modalities on a variety of hyperspectral imaging techniques. Specifically, we demonstrate feature segmentation and classification on the Indian Pines hyperspectral dataset and simultaneous location prediction of multiple drugs in mass spectrometry imaging of rat liver tissue. We further demonstrate label-free fluorescence image prediction from hyperspectral stimulated Raman scattering microscopy images. The applicability of the U-within-U-Net architecture on diverse datasets with widely varying input and output dimensions and data sources suggest that it has great potential in advancing the use of hyperspectral imaging across many different application areas ranging from remote sensing, to medical imaging, to microscopy.

In vivo simultaneous nonlinear absorption Raman and fluorescence (SNARF) imaging of mouse brain cortical structures

“…Both SNR and resolution can be improved through aberration correction as described elsewhere 45 . Deep learning can potentially replace machine learning to offer better performance in cell prediction 46 . In combination with genetic labeling of different subtypes of neurons, SNARF will expand the features that can be monitored simultaneously and provide a powerful platform for in vivo imaging of the structure and function of the brain cortex.…”

In Vivo Simultaneous Nonlinear Absorption Raman and Fluorescence (SNARF) Imaging of Mouse Brain Cortical Structures

“…60 Deep learning can potentially replace machine learning to offer better performance in cell prediction. 61 In combination with genetic labeling of different subtypes of neurons, SNARF significantly expands the features that can be monitored simultaneously and provides a powerful platform for in vivo imaging of the structure and function of the brain cortex.…”

Fast, live-cell imaging of 15 intracellular compartments by deep learning segmentation of super-resolution data

“…The long duration, as a result of the multiple excitations and image acquisition steps, is essentially required in multi-color fluorescence imaging, and consequently the phototoxicity further becomes a concern in live-cell imaging. Deep learning has been recently introduced to in-silico prediction of multi-color images from transmitted bright field images [6][7][8][9] . Nevertheless, they are all spatially diffraction-limited, suffering from low resolution and low contrast, therefore the prediction accuracy in recognizing the intracellular organelles is yet to be satisfactory 6 .…”