Virtual Staining of Defocused Autofluorescence Images of Unlabeled Tissue Using Deep Neural Networks

Abstract

Deep learning-based virtual staining was developed to introduce image contrast to label-free tissue sections, digitally matching the histological staining, which is time-consuming, labor-intensive, and destructive to tissue. Standard virtual staining requires high autofocusing precision during the whole slide imaging of label-free tissue, which consumes a significant portion of the total imaging time and can lead to tissue photodamage. Here, we introduce a fast virtual staining framework that can stain defocused autofluorescence images of unlabeled tissue, achieving equivalent performance to virtual staining of in-focus label-free images, also saving significant imaging time by lowering the microscope’s autofocusing precision. This framework incorporates a virtual autofocusing neural network to digitally refocus the defocused images and then transforms the refocused images into virtually stained images using a successive network. These cascaded networks form a collaborative inference scheme: the virtual staining model regularizes the virtual autofocusing network through a style loss during the training. To demonstrate the efficacy of this framework, we trained and blindly tested these networks using human lung tissue. Using 4× fewer focus points with 2× lower focusing precision, we successfully transformed the coarsely-focused autofluorescence images into high-quality virtually stained H&E images, matching the standard virtual staining framework that used finely-focused autofluorescence input images. Without sacrificing the staining quality, this framework decreases the total image acquisition time needed for virtual staining of a label-free whole-slide image (WSI) by ~32%, together with a ~89% decrease in the autofocusing time, and has the potential to eliminate the laborious and costly histochemical staining process in pathology.

Deep learning-assisted low-cost autofluorescence microscopy for rapid slide-free imaging with virtual histological staining

“…Independent training of the two cascaded models from super-resolution to virtual staining may also have a potential downside of accumulation of errors. Apart from obtaining higher resolution images as training targets, other potential improvements on the model training perspective could be using a super-resolution model with better image degradation modeling design [ 32 ], and using end-to-end training for the cascaded model [ 33 ].…”

Digital staining facilitates biomedical microscopy

“…For instance, it has been shown that GAN networks can be constructed and trained to deblur out-of-focus images with high reliability in frames with axial offsets of up to +/− 5 µm from the image plane ( Luo et al, 2021 ). To accelerate the tissue imaging process, which often consists of frequent focus adjustments during the scanning of a WSI, cascaded networks have been assembled to first restore the sharpness of defocused images that randomly appear during the slide scanning process, and then digitally perform virtual staining on these autofocused images ( Zhang Y. et al, 2022 ). This two-step tactic enabled by a cascade of autofocusing and virtual staining neural networks is an example of how deep learning can be used to enhance not only the sample preparation and staining processes, but also the measurement, i.e., the image acquisition step.…”

Virtual histological staining of unlabeled autopsy tissue

“…To mitigate these issues, a much finer image registration process using elastic registration algorithms can be employed to achieve pixel-level alignment between the paired images. This step is typically utilized in training image data preparation for supervised learning of virtual staining models for biopsy tissue samples 12,32,42 . However, due to its iterative and intense computational nature, such an elastic registration process is generally highly time-consuming and requires substantial data storage.…”

“…Network R compares the virtually stained images outputted by network G and their histochemically stained corresponding ground truth, coarsely registered using affine transformation-based registration steps, and it rapidly outputs a displacement vector field (DVF) that characterizes the pixel-wise relative displacement between the two images. Compared to elastic image registration methods that are based on iterative multi-scale image cross correlations 12,22,32,42 , network R substantially shortens the time required for the training image registration process 43 . In each batch of the learning, the resulting DVF is further fed into a spatial transformation module, which consequently aligns the histochemically stained ground truth images based on the DVF to ensure precise registration with the output of the network G (see the Methods); this process dynamically corrects and aligns the training image targets for the network G. As the training progresses, these networks improve their respective capabilities, which can be attributed to frequently alternating iterations between the training of the networks.…”