Digital staining through the application of deep neural networks to multi-modal multi-photon microscopy

Abstract

Deep neural networks have been used to map multi-modal, multi-photon microscopy measurements of a label-free tissue sample to its corresponding histologically stained brightfield microscope colour image. It is shown that the extra structural and functional contrasts provided by using two source modes, namely two-photon excitation microscopy and fluorescence lifetime imaging, result in a more faithful reconstruction of the target haematoxylin and eosin stained mode. This modal mapping procedure can aid histopathologists, since it provides access to unobserved imaging modalities, and translates the high-dimensional numerical data generated by multi-modal, multi-photon microscopy into traditionally accepted visual forms. Furthermore, by combining the strengths of traditional chemical staining and modern multi-photon microscopy techniques, modal mapping enables label-free, non-invasive studies of in vivo tissue samples or intravital microscopic imaging inside living animals. The results show that modal co-registration and the inclusion of spatial variations increase the visual accuracy of the mapped results.

Deep-Learning-Based Image Reconstruction and Enhancement in Optical Microscopy

“…Recently, deep learning has been demonstrated to be an effective virtual staining method. The success of this virtual staining has been illustrated using several different microscopy modalities as input images [49], [51], [126], [127]. Rivenson et al [49] demonstrated the efficacy of the deep learning-based staining using images of a single autofluorescence channel of an unstained tissue and that the technique can be applied to several different tissue types and stains (Masson's Trichrome and Jones stain in addition to the standard H&E).…”

HISTOBREAST, a collection of brightfield microscopy images of Haematoxylin and Eosin stained breast tissue

“…Image registration is the process of bringing in the same system of coordinates, and geometrically aligning, two or more images of the same scene, which may be taken at different times, under different acquisition conditions, or by different sensors 35 . In microscopy imaging, multimodal and/or monomodal registration are typically required to perform side-by-side image comparison, which is often needed to monitor morphological changes and motion, or to perform template-, or atlas-based segmentation, classification, detection, etc [36][37][38][39][40][41] . Furthermore, monomodal registration, e.g.…”

Preliminary Evaluation of the Utility of Deep Generative Histopathology Image Translation at a Mid-Sized NCI Cancer Center

“…A growing collection of studies have used GANs to synthetically stain images of histological tissue sections, which can save institutions time and money (both in reagents and technologists' time) (Bayramoglu et al, 2017;Borhani et al, 2019;De Biase, 2019;Lahiani et al, 2018;Quiros et al, 2019;Rana et al, 2018;Rivenson, Liu, et al, 2019;Rivenson, Wang, et al, 2019;Xu et al, 2019). GAN models have also been used to remove artificial and natural discolorations in images of stained histological tissue sections, removing artifacts that could perturb deep learning analyses (Bentaieb & Hamarneh, 2018;Ghazvinian Zanjani et al, 2018;Pontalba et al, 2019).…”