Structure-Preserving Multi-domain Stain Color Augmentation Using Style-Transfer with Disentangled Representations

Wagner, Sophia J.; Khalili, Nadieh; Sharma, Rishu; Boxberg, Melanie; Marr, Carsten; Back, Walter de; Peng, Tingying

doi:10.1007/978-3-030-87237-3_25

Cited by 32 publications

(13 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several classes of color normalization attempt to address this issue: (1) template color matching maps summary statistics or histograms of RGB values in a reference image to those in a reference image (Reinhard et al 2001; Kothari et al 2011; Y.-Y. Wang et al 2007; Magee et al 2009; Janowczyk, Basavanhally, and Madabhushi 2017); (2) color deconvolution (Ruifrok and Johnston 2001) represents each of the H&E dyes as a “stain vector” and substitutes a reference stain vector for the corresponding target vector (Macenko et al 2009; Rabinovich, A. and Agarwal, S. and Laris, C. A. and Price, J. H. and Belongie, S. n.d.; Trahearn et al 2015; Magee et al 2009; Khan et al 2014; Shafiei et al 2020; Salvi, Michielli, and Molinari 2020; Zheng et al 2019; Vahadane et al 2016); and (3) generative adversarial network (GAN) approaches transfer strain distributions from a reference dataset (rather than single image) to a target dataset (Bentaieb and Hamarneh 2018; Tarek Shaban et al 2018; Wagner et al 2021). Our repository has considerable variability across the dimensions impacting image color, with six sites contributing data generated at two resolutions (20X and 40X) from different scanner models across multiple years.…”

Section: Discussionmentioning

confidence: 99%

A pan-cancer PDX histology image repository with genomic and pathological annotations for deep learning analysis

White

Woo

Koc

et al. 2022

Preprint

View full text Add to dashboard Cite

Patient-derived xenografts (PDXs) model human intra-tumoral heterogeneity in the context of the intact tissue of immunocompromised mice. Histological imaging via hematoxylin and eosin (H&E) staining is performed on PDX samples for routine assessment and, in principle, captures the complex interplay between tumor and stromal cells. Deep learning (DL)-based analysis of large human H&E image repositories has extracted inter-cellular and morphological signals correlated with disease phenotype and therapeutic response. Here, we present an extensive, pan-cancer repository of nearly 1,000 PDX and paired human progenitor H&E images. These images, curated from the PDXNet consortium, are associated with genomic and transcriptomic data, clinical metadata, pathological assessment of cell composition, and, in several cases, detailed pathological annotation of tumor, stroma, and necrotic regions. We demonstrate that DL can be applied to these images to classify tumor regions and to predict xenograft-transplant lymphoproliferative disorder, the unintended outgrowth of human lymphocytes at the transplantation site. This repository enables PDX-specific, investigations of cancer biology through histopathological analysis and contributes important model system data that expand on existing human histology repositories. We expect the PDXNet Image Repository to be valuable for controlled digital pathology analysis, both for the evaluation of technical issues such as stain normalization and for development of novel computational methods based on spatial behaviors within cancer tissues.

show abstract

Section: Discussionmentioning

confidence: 99%

A pan-cancer PDX histology image repository with genomic and pathological annotations for deep learning analysis

White

Woo

Koc

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Previous works on style transfer mainly focus on designing advanced generative adversarial networks (GAN) for improved image diversity and fidelity under the assumption that a large-scale dataset with diverse image styles is available [10,11]. Recent findings show that domain shift is closely related to image style changes across different domains [9,12] and can be alleviated by increasing the diversity of training image styles [9,[13][14][15][16][17]]. One such successful example is MixStyle [9], which generates 'novel' styles via simply linearly mixing style statistics from two arbitrary training instances from the same domain at feature level.…”

Section: Related Workmentioning

confidence: 99%

MaxStyle: Adversarial Style Composition for Robust Medical Image Segmentation

Chen¹,

Li²,

Ouyang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) have achieved remarkable segmentation accuracy on benchmark datasets where training and test sets are from the same domain, yet their performance can degrade significantly on unseen domains, which hinders the deployment of CNNs in many clinical scenarios. Most existing works improve model out-of-domain (OOD) robustness by collecting multi-domain datasets for training, which is expensive and may not always be feasible due to privacy and logistical issues. In this work, we focus on improving model robustness using a single-domain dataset only. We propose a novel data augmentation framework called MaxStyle, which maximizes the effectiveness of style augmentation for model OOD performance. It attaches an auxiliary style-augmented image decoder to a segmentation network for robust feature learning and data augmentation. Importantly, MaxStyle augments data with improved image style diversity and hardness, by expanding the style space with noise and searching for the worst-case style composition of latent features via adversarial training. With extensive experiments on multiple public cardiac and prostate MR datasets, we demonstrate that MaxStyle leads to significantly improved out-ofdistribution robustness against unseen corruptions as well as common distribution shifts across multiple, different, unseen sites and unknown image sequences under both low-and high-training data settings. The code can be found at https://github.com/cherise215/MaxStyle.

show abstract

“…Style transfer has been widely used for image synthesis from one modality to another, e.g., from computed tomography (CT) to magnetic resonance imaging (MRI) [ 6 ], from MRI to CT [ 7 ], from MRI to Positron Emission Tomography (PET) [ 8 ]. It has also been shown to be effective for histological images, e.g., for stain color augmentation [ 9 ]. In terms of style transfer architectures, conditional GAN (cGAN) [ 10 ] can be used for aligned image pairs and its generator can generate images of a certain class.…”

Section: Introductionmentioning

confidence: 99%

Multi-Modality Microscopy Image Style Augmentation for Nuclei Segmentation

Liu

Wagner

Peng³

2022

J. Imaging

Self Cite

View full text Add to dashboard Cite

Annotating microscopy images for nuclei segmentation by medical experts is laborious and time-consuming. To leverage the few existing annotations, also across multiple modalities, we propose a novel microscopy-style augmentation technique based on a generative adversarial network (GAN). Unlike other style transfer methods, it can not only deal with different cell assay types and lighting conditions, but also with different imaging modalities, such as bright-field and fluorescence microscopy. Using disentangled representations for content and style, we can preserve the structure of the original image while altering its style during augmentation. We evaluate our data augmentation on the 2018 Data Science Bowl dataset consisting of various cell assays, lighting conditions, and imaging modalities. With our style augmentation, the segmentation accuracy of the two top-ranked Mask R-CNN-based nuclei segmentation algorithms in the competition increases significantly. Thus, our augmentation technique renders the downstream task more robust to the test data heterogeneity and helps counteract class imbalance without resampling of minority classes.

show abstract

Structure-Preserving Multi-domain Stain Color Augmentation Using Style-Transfer with Disentangled Representations

Cited by 32 publications

References 16 publications

A pan-cancer PDX histology image repository with genomic and pathological annotations for deep learning analysis

A pan-cancer PDX histology image repository with genomic and pathological annotations for deep learning analysis

MaxStyle: Adversarial Style Composition for Robust Medical Image Segmentation

Multi-Modality Microscopy Image Style Augmentation for Nuclei Segmentation

Contact Info

Product

Resources

About