Virtualization of Tissue Staining in Digital Pathology Using an Unsupervised Deep Learning Approach

Abstract

Histopathological evaluation of tissue samples is a key practice in patient diagnosis and drug development, especially in oncology. Historically, Hematoxylin and Eosin (H&E) has been used by pathologists as a gold standard staining. However, in many cases, various target specific stains, including immunohistochemistry (IHC), are needed in order to highlight specific structures in the tissue. As tissue is scarce and staining procedures are tedious, it would be beneficial to generate images of stained tissue virtually. Virtual staining could also generate in-silico multiplexing of different stains on the same tissue segment. In this paper, we present a sample application that generates FAP-CK virtual IHC images from Ki67-CD8 real IHC images using an unsupervised deep learning approach based on CycleGAN. We also propose a method to deal with tiling artifacts caused by normalization layers and we validate our approach by comparing the results of tissue analysis algorithms for virtual and real images.

Preliminary Evaluation of the Utility of Deep Generative Histopathology Image Translation at a Mid-Sized NCI Cancer Center

“…A growing collection of studies have used GANs to synthetically stain images of histological tissue sections, which can save institutions time and money (both in reagents and technologists' time) (Bayramoglu et al, 2017;Borhani et al, 2019;De Biase, 2019;Lahiani et al, 2018;Quiros et al, 2019;Rana et al, 2018;Rivenson, Liu, et al, 2019;Rivenson, Wang, et al, 2019;Xu et al, 2019). GAN models have also been used to remove artificial and natural discolorations in images of stained histological tissue sections, removing artifacts that could perturb deep learning analyses (Bentaieb & Hamarneh, 2018;Ghazvinian Zanjani et al, 2018;Pontalba et al, 2019).…”

PathFlow-MixMatch for Whole Slide Image Registration: An Investigation of a Segment-Based Scalable Image Registration Method

“…molecular alterations, clinical chemistry, survival, etc. ), via corresponding immunohistochemistry stains (IHC), 5,6 and mutational panels of known oncological driver mutations (among others) [7][8][9] . Furthermore, generative techniques have been developed to computationally translate one histological stain (e.g.…”

Perceptual Embedding Consistency for Seamless Reconstruction of Tilewise Style Transfer

“…arXiv:1906.00617v1 [cs.CV] 3 Jun 2019 [7,12]. Deep learning based style transfer has been used to generate augmented faces [3], virtual artwork with specific artist styles [16] and recently also virtually stained histopathological whole slide images (WSIs) [8,4]. Some groups used approximative and empirical methods in order to virtually generate H&E images from fluorescence images [6,9].…”

“…The training of these supervised methods is based on spatially registered image pairs of the input and output modalities. As generating paired slide images with different stainings is a complex task involving the use of consecutive tissue sections or a stain-washstain technique, unsupervised deep learning methods have been used in virtual staining [8] and stain normalization applications [14]. In [8], CycleGAN [16] has been used in order to virtually generate duplex Immunohistochemistry (IHC) stained images from real stained images.…”

“…For this reason, tile based analysis is usually used in order to deal with memory limitations. Inference of trained generators on WSIs in a tilewise manner when instance normalization modules are used has been shown to present tiling artifacts in the reconstructed WSI [8]. While instance normalization has been proven to be crucial in style transfer applications GAN training [15], it makes a pixel in the output image depend not only on the network and the receptive field area but also on the statistics of the entire input image.…”