Detection and classification of neurons and glial cells in the MADM mouse brain using RetinaNet

Abstract

The ability to automatically detect and classify populations of cells in tissue sections is paramount in a wide variety of applications ranging from developmental biology to pathology. Although deep learning algorithms are widely applied to microscopy data, they typically focus on segmentation which requires extensive training and labor-intensive annotation. Here, we utilized object detection networks (neural networks) to detect and classify targets in complex microscopy images, while simplifying data annotation. To this end, we used a RetinaNet model to classify genetically labeled neurons and glia in the brains of Mosaic Analysis with Double Markers (MADM) mice. Our initial RetinaNet-based model achieved an average precision of 0.90 across six classes of cells differentiated by MADM reporter expression and their phenotype (neuron or glia). However, we found that a single RetinaNet model often failed when encountering dense and saturated glial clusters, which show high variability in their shape and fluorophore densities compared to neurons. To overcome this, we introduced a second RetinaNet model dedicated to the detection of glia clusters. Merging the predictions of the two computational models significantly improved the automated cell counting of glial clusters. The proposed cell detection workflow will be instrumental in quantitative analysis of the spatial organization of cellular populations, which is applicable not only to preparations in neuroscience studies, but also to any tissue preparation containing labeled populations of cells.

COMBINe: Automated Detection and Classification of Neurons and Astrocytes in Tissue Cleared Mouse Brains

“…Both the training data and the validation data contained fluorescence images from different imaging modalities, e.g., slide scanner, confocal fluorescence microscopy, and LSFM (Y. Cai et al, 2021). 3361 individual cells (2497 neurons and 864 astrocytes) were labeled in the training data.…”

“…Data augmentation strategies such as geometrical transforms, color swap and saturation simulation were applied (Y. Cai et al, 2021). The initial learning rate was 0.0001 and the batch size was four.…”

“…We explored the training regime of the network and applied it to fluorescence images of MADM brain sections in a pervious study (Y. Cai et al, 2021). We again employed the data augmentation of color swap and saturation to balance the color classes and utilized transfer learning to train the network on a pretrained backbone.…”

“…Inspired with these studies, we have surveyed the deep learning literature, and employ a deep learning model (RetinaNet) that show superior performance in dense object detection and apply it to a neurodevelopmental problem (Y. Cai et al, 2021;Lin et al, 2018;Waithe et al, 2020).…”

“…In combination with tissue clearing and 3D microscopy, deep learning has also been utilized to study cancer metastasis in mice (Pan et al, 2019), segment mouse brain vasculature (Todorov et al, 2020), and detect crown-like structures in adipose tissues (Geng et al, 2021). Inspired with these studies, we have surveyed the deep learning literature, and employ a deep learning model (RetinaNet) that show superior performance in dense object detection and apply it to a neurodevelopmental problem (Y. Cai et al, 2021; Lin et al, 2018; Waithe et al, 2020).…”